Introduction to Machine Learning

Virginia Tech, Electrical and Computer Engineering
Spring 2015: ECE 5984

Course Information


Have you ever wondered how Siri understands voice commands? How Netflix recommends movies to watch? How Kinnect recognizes full-body gestures? How Watson defeated Jeopardy champions? The answer is Machine Learning -- the study of algorithms that learn from large quantities of data, identify patterns and make predictions on new instances.

Instructor
Dhruv Batra
Office Hour
Fri 3-4pm, Whittemore 468

TA

Qing Sun (sunqing -at- vt)
Office Hour
Tue 5-6pm, Whittemore 415

Class meets

Mon, Wed 4:00 - 5:15, GBJ 102

Homeworks

Q&A and Discussions
https://scholar.vt.edu/portal/site/s15ece5984
ECE5984 Forum
Class Kaggle Links
http://inclass.kaggle.com/c/VT-ECE-Machine-Learning-HW1
http://inclass.kaggle.com/c/2015-Spring-vt-ece-machine-learning-hw2
http://inclass.kaggle.com/c/2015-Spring-vt-ece-machine-learning-hw3

HW Kaggle Challenge
Curators

Faruk Ahmed, Aroma Mahendru,
Senthil Purushwalkam
This course introduces the basics of Machine Learning:
  • Supervised Learning (Classification/Regression)
    • Nearest Neighbour
    • Naive Bayes
    • Logistic Regression
    • Support Vector Machine
    • Neural Networks (Multi-layer Perceptrons, New this year: Convolutional Neural Networks, Recurrrent Neural Nets)
    • Decision Trees
  • Unsupervised Learning
    • Clustering, Kmeans
    • Dimensionality Reduction (Principal Component Analysis, Linear Discriminant Analysis)
  • Other Learning Setting and Advanced Topics
    • Weakly-Supervised and Semi-Supervised Learning
    • Reinforcement Learning
    • Learning Theory (VC dimensions, generalization error bounds, PAC Theory)
    • Structured Learning (Bayes Nets, Markov Random Fields, Structured SVMs)

Best Project Prize Winners


  • Winning Team:

    Latha Pemula.
    Low Power Image Recognition. (Contribution: Speeding up Convolutional Neural Networks by using basis filters)

  • Runners-up:

    Kexiong (Curtis) Zeng, Jinshan Liu.
    Image-based Localization System for Mobile Devices.

    Venkata Pavan Kumar Rangudu, Elaheh Raisi.
    Digit Recognition.

Notable Mentions

Here are some great achievements by the students who took this class:

  • Kexiong (Curtis) Zeng's class project accepted for publication at MobiSys.

    Poster: Location Verification and Recovery for Mobile In-Vehicle Applications.
    Kexiong (Curtis) Zeng, Yanzhi Dou, Yaling Yang and Ranveer Chandra.
    In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys), 2015.

  • If you took this class and have something to share, please email me!


Schedule (Tentative)

Week/Day Date Topic Notes
1 W Jan 21 Overview of Machine Learning
Slides (pptx), Slides (pdf), Video
HW0 is out. Due: Jan 23.
Readings: Barber Chap 1, 13.1.
[Optional] Video: Sam Roweis -- What is Machine Learning?
2 M Jan 26 Supervised Learning
  • General Setup: learning from data
  • Your First Classifier: 1/k Nearest Neighbour
Slides (pptx), Slides (pdf), Notes

Reading: Barber Chap 14.
[Optional] Video: Pedro Domingos -- The K-NN Algorithm
2 W Jan 28
  • (Finish) 1/k Nearest Neighbour
  • Kernel regression/classification
Slides (pptx), Slides (pdf), Notes
3 M Feb 2
  • (Finish) Kernel regression/classification
  • Curse of Dimensionality
Slides (pptx), Slides (pdf), Notes

HW1 is out. Due: Feb 15.
3 W Feb 4 Project Details
Supervised Statistical Learning
  • Probability Review
Slides (pptx), Slides (pdf), Notes

Readings: Barber 8.1, 8.2.
[Optional] Videos: Probability Primer.
[Optional] Video: Michael Jordon -- Bayesian or Frequentist: Which Are You?
4 M Feb 9
  • Statistical Estimation (MLE, MAP, Bayesian)
Slides (pptx), Slides (pdf), Notes

[Optional] Video: Daphne Koller -- Coursera: Probabilistic Graphical Models, MLE Lecture, MAP Lecture.
Readings: Barber 8.6, 8.7.
4 W Feb 11
  • Gaussians
  • Linear Models
    • Regression
Slides (pptx), Slides (pdf), Notes

Readings: Barber 8.4, 17.1, 17.2.
5 M Feb 16 VT closed due to inclement weather. No class.
5 W Feb 18
  • Regression
Slides (pptx), Slides (pdf), Notes

HW2 is out. Due: Mar 6.
Readings: 17.1, 17.2.
6 M Feb 23
  • (Finish) Regression
  • Model Selection
  • Error Decomposition
  • Cross-validation
Slides (pptx), Slides (pdf), Notes

Project proposal is due: Feb 24.
6 W Feb 25
  • (Finish) Error Decomposition
    • Bias-Variance Tradeoff
  • Classification: Naive Bayes
Slides (pptx), Slides (pdf),

Readings: Barber 10.1-3.
[Optional] Video: Andrew Ng -- Naive Bayes
7 M Mar 2
  • (Finish) Naive Bayes
  • Classification: Logistic Regression
Slides (pptx), Slides (pdf), Notes

7 W Mar 4
  • (Finish) Logistic Regression
  • NB & LR connections
Slides (pptx), Slides (pdf), Notes

Readings: Barber 17.4.
[Optional] Video: Andrew Ng -- Logistic Regression.
Reading: Tom Mitchell -- Book Chapter: Naive Bayes and Logistic Regression
8 M Mar 9 Spring Break: No Class.
8 W Mar 11 Spring Break: No Class.
9 M Mar 16 No Class. Convex Optimization video lectures. Readings: Boyd and Vandenberghe Chap 2 (Convex Sets): 2.1, 2.2, 2.3; Chap 3 (Convex Functions): 3.1, 3.2.
[Optional] Video: Stephen Boyd -- Convex Optimization.
9 W Mar 18 In class Mid-Term
10 M Mar 23
  • Support Vector Machines
    • Linear SVMs
    • Hard & soft-margin SVMs
Slides (pptx), Slides (pdf), Notes

Readings: Barber 17.5.
[Optional] Video: Andrew Ng -- KKT Conditions and SVM Duality.
[Optional] Video: Stephen Boyd -- Lagrangian Duality.
10 W Mar 25
  • Support Vector Machines
    • Lagrangian Duality
    Slides (pptx), Slides (pdf), Notes

11 M Mar 30
  • Support Vector Machines
    • SVM dual
    • Kernel Trick
Slides (pptx), Slides (pdf), Notes

HW3 is out.
11 W Apr 1
  • (Finish) Support Vector Machines
    • Multi-class SVM
  • Neural Networks
    • Perceptron
    • Multilayer Perceptron
Slides (pptx), Slides (pdf), Notes

Reading: Murphy 16.5.
[Optional] Reading: Hastie, Tibshirani, Friedman -- Chap 11.
[Optional] Video: Andrew Ng -- Coursera: Machine Learning,
Neural Networks lecture, Backpropagation lecture.
12 M Apr 6
  • Neural Networks
    • Backprop
    • Autoencoders
    • Convolutional Nets
Slides (pptx), Slides (pdf), Notes

Reading: Murphy 16.5.
[Optional] Reading: Hastie, Tibshirani, Friedman -- Chap 11.
[Optional] Reading: Yann LeCun et al. -- Efficient BackProp
12 W Apr 8
  • Decision Trees
    • Mutual Information
Slides (pptx), Slides (pdf), Notes

Reading: Murphy 16.1-16.2. [Optional] Reading: Hastie, Tibshirani, Friedman -- Chap 9.2.
[Optional] Video: Pedro Domingos -- Coursera: Machine Learning,
Decision Trees lecture,
13 M Apr 13
  • (Finish) Decision Trees
Ensemble Methods
  • Bagging
Slides (pptx), Slides (pdf)

13 W Apr 15 Ensemble Methods
  • (Finish) Bagging
  • Boosting
Learning Theory
  • PAC-Learning
  • Generalization bounds
Slides (pptx), Slides (pdf), Notes

Reading: Murphy 16.4.
[Optional] Video: Robert Schapire -- Boosting
[Optional] Reading: Tom Dietterich -- Ensemble Methods in Machine Learning.
14 M Apr 20 In-class project presentations. HW4 is out.
14 W Apr 22 ICCV deadline. No class.
15 M Apr 27 Unsupervised Learning
  • Clustering
    • K-Means
  • Gaussian Mixture Models
Slides (pptx), Slides (pdf) Notes

Readings: Barber 20.1-20.3.
[Optional] Video: Andrew Ng -- Clustering, GMM
15 W Apr 29
  • EM for GMMs
  • EM for general unsupervised learning
Slides (pptx), Slides (pdf) Notes

16 M May 4
  • (Finish) EM for general unsupervised learning
  • Factor Analysis
    • PCA
    • SVD
    • Dimensionality Reduction
Slides (pptx), Slides (pdf), Notes

Readings: Barber 15.1-15.4.
[Optional] Video: Andrew Ng -- PCA
16 W May 6 Overview of Advanced Topics
  • Probabilistic Graphical Models
    • Bayes Nets
    • Hidden Markov Models (HMMs)
  • Online Learning
  • Active Learning
  • Semi-Supervised Learning
  • Reinforcement Learning

[Optional] Reading: Pedro Domingo -- A Few Useful Things to Know about Machine Learning
[Optional] Video: Andrew Ng -- Advice for Applying Machine Learning
May 8 Project Poster+Demo Presentation 310 Kelly Hall; 1:30 - 4:00 pm
May 11 Final Exam In class (GBJ 102); 7:45 - 9:45 am

Grading

  • 40% Homework
  • 25% Project
  • 10% Mid Term
  • 20% Final
  • 5% Class Participation

Late policy for deliverables

  • No penalties for medical reasons (bring doctor's note) or emergencies.
  • Every student has 5 free late days (5 x 24hour-chunks) for this course; these cannot be used for exams (midterm or final) or the poster session.
  • After all free late days are used up, penalty is 25% for each additional late day.

Collaboration Policy

  • Study groups are allowed. You can discuss the questions and potential directions for solving them. However, your discussions should be collaborative and not involve one student simply telling the other student the solutions.
  • If you worked in a group, please write the names of the other participants on the first page of your assignment.
  • Important: you need to write your solutions and code independently, and not as a group activity.
  • We often reuse problem set questions from various other sources. We expect students to want to learn and not google for answers. The purpose of problem sets in this class is to help you think about the material, not just give us the right answers. We take Honor Code violations seriously.

Project

Team project: 1-3 people. Description and grading policy (proposal, midsem presentation, final poster + demo presentation).

Textbooks

None required. Reference Books:

[On Library Reserve]
Machine Learning: A Probabilistic Perspective, Kevin Murphy

[Free PDF from the book webpage]
The Elements of Statistical Learning, Hastie, Tibshirani, and Friedman

[Free PDF from author's webpage]
Bayesian Reasoning and Machine Learning, David Barber

[Available in the Library]
Pattern Recognition and Machine Learning, Chris Bishop

Prerequisites

  • Probability and Statistics
    • Distributions, densities, marginalization, moments, typical distributions (Gaussian, Laplacian, etc)
  • Calculus and Linear Algebra
    • Matrix multiplication, eigenvalues, positive semi-definiteness, multivariate derivates.
  • Algorithms
    • Dynamic programming, basic data structures, complexity (NP-hardness).
  • Programming
    • Matlab for HWs. Your language of choice for project.
    • No coding / compilation / debugging support.
  • Ability to deal with abstract mathematical concepts.

Related Classes / Online Resources

  • 10-701 Machine Learning, Carnegie Mellon University
  • CIS 520 Machine Learning, UPenn
  • CS 229 Machine Learning, Stanford
  • CSE 546 Machine Learning, University of Washington
  • Machine Learning, Coursera

Note to people outside VT

Feel free to use the slides and materials available online here.
If you use our slides, an appropriate attribution is requested.
Please email the instructor with any corrections or improvements.

© 2013 Virginia Tech

Webpage CSS courtesy Bootstrap and Polo Chau.