Deep Learning for Perception
Virginia Tech, Electrical and Computer Engineering
Fall 2015: ECE 6504
Course Information
This is an exciting time to be studying (Deep) Machine Learning, or Representation Learning, or for lack of a better term, simply Deep Learning!
Deep Learning is rapidly emerging as one of the most successful and widely applicable set of techniques across a range of applications (vision, language, speech, computational biology, robotics, etc), leading to some pretty significant commercial success.
This course will expose students to cuttingedge research — starting from a refresher in basics of neural networks, to recent developments. The emphasis will be on studentled paper presentations and discussions. Each “module” will begin with instructor lectures to present context and background material.
NOTE: Only the lectures delivered by the course instructor are made public. We regret that student presentations cannot be made public due to privacy reasons.
 Instructor
 Dhruv Batra
 Office Hour
 Fri 34pm, Whittemore 468
 Teaching Assistants
 Abhishek Das
Ashwin Kalyan  Office Hours
 Ashwin Kalyan: Mon 34pm, Whittemore 415
Abhishek Das: Fri 34pm, Whittemore 415  Class meets
 Tue, Thu 5:00  6:15, SURGE 107
 Scholar Site
 https://scholar.vt.edu/portal/site/f15ece6504
 Q&A and Discussions
 Scholar Discussion Forum
 Staff Mailing List
 stafff15ece6504g@vt.edu
Schedule
Date  Topic  Primary Reading  Optional Reading (HR = Historically Relevant) 
Video  
W1: Aug 25 
Class Administrativia Recall: Logistic Regression, MLP, Maximum Likelihood, Gradient Descent Slides (pptx), Slides (pdf). 

W1: Aug 27 
Introduction to Caffe Slides (pptx), Slides (pdf). 

W2: Aug 31  Homework 0 due date  
W2: Sep 1 
Lecture: Neural Networks, Activation Functions, Backpropagation Slides (pptx), Slides (pdf), Notes (pdf). 

W2: Sep 3 
Lecture: ConvNets Slides (pptx), Slides (pdf), Notes (pdf). 

Module 1  
W3: Sep 8 
ConvNets [Presenter: Harsh] Dhruv Slides (pptx), Dhruv Slides (pdf), Dhruv Notes (pdf), Harsh Slides (pptx). 


W3: Sep 10 
Regularization [Presenters: Xiao Lin & Peng] Slides (ppt). 

W4: Sep 15 
Finetuning [Presenters: Delasa & Abraham] Slides (ppt). 

W4: Sep 17 
Deeper is Better [Presenter: Latha P] Notes (ppt). Slides (ppt). 


W5: Sep 22  Homework 1 due date  
W5: Sep 22 
Visualizing ConvNets [Presenters: Abhijit Sarkar & Tamoghna] Notes (ppt), Slides(pdf) , Slides(pdf) . 


W5: Sep 24 
Fooling ConvNets and Adversarial Examples [Presenter: Aroma] Slides(ppt) . 

W6: Sep 29 
Robustness to Adversarial Examples [Presenters: Pooja & Daniel Friedman] Slides(ppt) . 

W6: Oct 1 
Detection ConvNets [Presenters: Viresh & Ardalan Khosrowpour] Slides(ppt) . 

W7: Oct 6  Homework 2 due date  
W7: Oct 6 
Segmentation ConvNets [Presenters: Gordon Christie & Mostafa] FCN Slides(ppt), FCN Slides(pdf), Zoomout, DeconvNet Slides(ppt). 

Module 2  
W7: Oct 8 
Lecture: Intro to RNNs & BackPropThroughTime (BPTT) Slides (pptx), Slides (pdf), Notes (pdf), Torch Tutorial. 

W8: Oct 13 
Character RNNs [Presenter: Ramprasaath] Slides (pptx), Slides (pdf). 

W8: Oct 15 
Different Units [Presenter: Rama] Slides(pptx) . 

Module 3  
W9: Oct 20 
Sequence to Sequence [Presenter: Haitham] Slides (pptx). 

W9: Oct 22 
CNNs+RNNs [Presenters: Yash & Azam Moosavi] Slides part1(pptx) Slides part2(pptx) . 

W10: Oct 26  Homework 3 due date  
W10: Oct 27 
Memory Networks [Presenter: Jiasen] Slides (pptx). 

W10: Oct 29 
Visual Question Answering (VQA) [Presenter: Aishwarya] 

W11: Nov 3  CVPR Deadline: No Class  
W11: Nov 5  CVPR Deadline: No Class  
Module 4  
W12: Nov 10 
Unsupervised [Presenters: Jianwei Yang & Stan Antol] Slides (pptx). 

W12: Nov 12 
Weakly supervised [Presenter: Igor Janjic] Slides(pdf) . 

W13: Nov 17 
Hyperparameters / Tweaking [Presenters: Yufeng & Chris Dusold] Batch Normalization, Initialization & Momentum slides(pdf), LR Optimizations slides(pdf). 

W13: Nov 19 
Ensembles [Presenter: Ujwal Krothapalli] Slides(ppt) . 

W14: Nov 24  Thanksgiving Break: No Class  
W14: Nov 26  Thanksgiving Break: No Class  
W15: Dec 1 
Bayesian NN [Presenters: Michael Cogswell & Qing] Weight Uncertainty slides (pptx), Dropout as Bayesian Approximation slides (pptx). 

W15: Dec 3 
Deep Reinforcement Learning [Presenter: Arjun] Slides(pdf) . 

W13: Dec 3  Homework 4 due date  
W16: Dec 8  NIPS 2015: No Class 
Grading
 40% Homework (4 homeworks x 10% each)
 15% Paper Presentation
 15% Paper Reviews
 25% Class Project
 5% Class Participation
Late policy for deliverables
 No penalties for medical reasons (bring doctor's note) or emergencies.
 Every student has 5 free late days (5 x 24hour chunks) for this course; these cannot be used for HW0, the reviews or the presentation.
 After all free late days are used up, penalty is 25% for each additional late day.
Prerequisites
ECE 6504 is an ADVANCED class. This should not be your first exposure to machine learning. Ideally, you need:
 Introlevel Machine Learning
 ECE 5984 or equivalent
 Algorithms
 Dynamic programming, basic data structures, complexity (NPhardness)
 Calculus and Linear Algebra
 positive semidefiniteness, multivariate derivates (be prepared for lots and lots of gradients!)
 Programming
 This is a demanding class in terms of programming skills.
 HWs will involve a mix of languages (Python, C++, and Lua) and libraries (Caffe and Torch).
 Your language of choice for project.
 Ability to deal with abstract mathematical concepts
FAQs
The class is full. Can I still get in?
See the advisors in 340 Whittemore Hall about forceadd forms, and come to the first class. There will be a HW0 released on day 1 to check background preparation. Our past experience with HW0 suggests that slots will open up after some students drop the class.
I have not taken an “Intro to Machine Learning” class or I am taking it in parallel. Can I still take this class?
Try working on HW0. If terms seem unfamiliar or you find yourself doing significant background reading to even understand the question, you may benefit from waiting for the next offering of this class.
Can I audit this class?
Depends. We will give preference to students wanting to take this class for credit, so if the class is full, there will be no audits.
Reviews
 Length <=1 page
 11 pt Times New Roman, 1 inch margins
 Due: Midnight before class
 Organization of Review:
 Summary:
 What is this paper about?
 What is the main contribution?
 Describe the main approach & results. Just facts, no opinions yet.
 Strengths:
 Is there a new theoretical insight?
 Or a significant empirical advance? Did they solve a standing open problem?
 Or is a good formulation for a new problem?
 Or a faster/better solution for an existing problem?
 Any good practical outcome (code, algorithm, etc)?
 Are the experiments well executed?
 Useful for the community in general?
 Weaknesses:
 What would you do differently?
 Any missing baselines? missing datasets?
 any odd design choices in the algorithm not explained well? quality of writing?
 Is there sufficient novelty in what they propose? Has it already been done? Minor variation of previous work?
 Why should anyone care? Is the problem interesting and significant?
 Reflections:
 How does this relate to other papers we have read?
 What are the next research directions in this line of work?
Presentations
 Once in the semester
 Expected to:
 read all papers listed for that day
 provide overview of that day’s theme
 present details of at least 2 papers and explain how different papers relate to each other.
 please clearly cite the source of each slide that is not your own.
 not submit paper review for that class
 meet with TA 34 days before class to dry run presentation (Worth 40% of presentation grade)
Projects
 You are encouraged to try out interesting applications of deep learning (vision, NLP,computational biology, UAVs, etc!)
 The project must be done in this semester. No double counting.
 You may combine with other course project but must delineate the different parts
 Extra credit for shooting a publication
 Main Categories:
 Application/survey  compare a bunch of algorithms on a new application domain of your interest
 Formulation/Development  Formulate a new model or algorithm for a new/od problem
 Theory: Theoretically analyze an existing algorithm.
Related Classes / Online Resources
 CS231n Convolutional Neural Networks for Visual Recognition, Stanford
 Machine Learning, Oxford
 Deep Learning, New York University
Book
 Deep Learning, Ian Goodfellow, Aaron Courville, and Yoshua Bengio, Book in preparation for MIT Press
Overviews
Note to people outside VT
Feel free to use the slides and materials available online here. If you use our slides, an appropriate attribution is requested. Please email the instructor with any corrections or improvements.
Acknowledgements
We thank NVIDIA for GPU donations and AWS Educate Grant for cloud computing resources.