Deep Learning for Perception

Virginia Tech, Electrical and Computer Engineering

Fall 2015: ECE 6504

Course Information

This is an exciting time to be studying (Deep) Machine Learning, or Representation Learning, or for lack of a better term, simply Deep Learning!

Deep Learning is rapidly emerging as one of the most successful and widely applicable set of techniques across a range of applications (vision, language, speech, computational biology, robotics, etc), leading to some pretty significant commercial success.

This course will expose students to cutting-edge research — starting from a refresher in basics of neural networks, to recent developments. The emphasis will be on student-led paper presentations and discussions. Each “module” will begin with instructor lectures to present context and background material.

NOTE: Only the lectures delivered by the course instructor are made public. We regret that student presentations cannot be made public due to privacy reasons.

Dhruv Batra
Office Hour
Fri 3-4pm, Whittemore 468

Teaching Assistants
Abhishek Das
Ashwin Kalyan
Office Hours
Ashwin Kalyan: Mon 3-4pm, Whittemore 415
Abhishek Das: Fri 3-4pm, Whittemore 415

Class meets
Tue, Thu 5:00 - 6:15, SURGE 107

Scholar Site
Q&A and Discussions
Scholar Discussion Forum
Staff Mailing List


Date Topic Primary Reading Optional Reading
(HR = Historically Relevant)
W1: Aug 25 Class Administrativia
Recall: Logistic Regression, MLP, Maximum Likelihood, Gradient Descent
Slides (pptx), Slides (pdf).
W1: Aug 27 Introduction to Caffe
Slides (pptx), Slides (pdf).
W2: Aug 31 Homework 0 due date
W2: Sep 1 Lecture: Neural Networks, Activation Functions, Backpropagation
Slides (pptx), Slides (pdf), Notes (pdf).
W2: Sep 3 Lecture: ConvNets
Slides (pptx), Slides (pdf), Notes (pdf).
Module 1
W3: Sep 8 ConvNets
[Presenter: Harsh]
Dhruv Slides (pptx), Dhruv Slides (pdf), Dhruv Notes (pdf), Harsh Slides (pptx).

W3: Sep 10 Regularization
[Presenters: Xiao Lin & Peng]
Slides (ppt).
W4: Sep 15 Finetuning
[Presenters: Delasa & Abraham]
Slides (ppt).
W4: Sep 17 Deeper is Better
[Presenter: Latha P]
Notes (ppt). Slides (ppt).

W5: Sep 22 Homework 1 due date
W5: Sep 22 Visualizing ConvNets
[Presenters: Abhijit Sarkar & Tamoghna]
Notes (ppt), Slides(pdf) , Slides(pdf) .

W5: Sep 24 Fooling ConvNets and Adversarial Examples
[Presenter: Aroma]
Slides(ppt) .
W6: Sep 29 Robustness to Adversarial Examples
[Presenters: Pooja & Daniel Friedman]
Slides(ppt) .
W6: Oct 1 Detection ConvNets
[Presenters: Viresh & Ardalan Khosrowpour]
Slides(ppt) .
W7: Oct 6 Homework 2 due date
W7: Oct 6 Segmentation ConvNets
[Presenters: Gordon Christie & Mostafa]
FCN Slides(ppt), FCN Slides(pdf), Zoom-out, DeconvNet Slides(ppt).
Module 2
W7: Oct 8 Lecture: Intro to RNNs & BackProp-Through-Time (BPTT)
Slides (pptx), Slides (pdf), Notes (pdf), Torch Tutorial.
W8: Oct 13 Character RNNs
[Presenter: Ramprasaath]
Slides (pptx), Slides (pdf).
W8: Oct 15 Different Units
[Presenter: Rama]
Slides(pptx) .
Module 3
W9: Oct 20 Sequence to Sequence
[Presenter: Haitham]
Slides (pptx).
W9: Oct 22 CNNs+RNNs
[Presenters: Yash & Azam Moosavi]
Slides part-1(pptx) Slides part-2(pptx) .
W10: Oct 26 Homework 3 due date
W10: Oct 27 Memory Networks
[Presenter: Jiasen]
Slides (pptx).
W10: Oct 29 Visual Question Answering (VQA)
[Presenter: Aishwarya]
W11: Nov 3 CVPR Deadline: No Class
W11: Nov 5 CVPR Deadline: No Class
Module 4
W12: Nov 10 Unsupervised
[Presenters: Jianwei Yang & Stan Antol]
Slides (pptx).
W12: Nov 12 Weakly supervised
[Presenter: Igor Janjic]
Slides(pdf) .
W13: Nov 17 Hyper-parameters / Tweaking
[Presenters: Yufeng & Chris Dusold]
Batch Normalization, Initialization & Momentum slides(pdf), LR Optimizations slides(pdf).
W13: Nov 19 Ensembles
[Presenter: Ujwal Krothapalli]
Slides(ppt) .
W14: Nov 24 Thanksgiving Break: No Class
W14: Nov 26 Thanksgiving Break: No Class
W15: Dec 1 Bayesian NN
[Presenters: Michael Cogswell & Qing]
Weight Uncertainty slides (pptx), Dropout as Bayesian Approximation slides (pptx).
W15: Dec 3 Deep Reinforcement Learning
[Presenter: Arjun]
Slides(pdf) .
W13: Dec 3 Homework 4 due date
W16: Dec 8 NIPS 2015: No Class


  • 40% Homework (4 homeworks x 10% each)
  • 15% Paper Presentation
  • 15% Paper Reviews
  • 25% Class Project
  • 5% Class Participation

Late policy for deliverables

  • No penalties for medical reasons (bring doctor's note) or emergencies.
  • Every student has 5 free late days (5 x 24-hour chunks) for this course; these cannot be used for HW0, the reviews or the presentation.
  • After all free late days are used up, penalty is 25% for each additional late day.


ECE 6504 is an ADVANCED class. This should not be your first exposure to machine learning. Ideally, you need:

  • Intro-level Machine Learning
  • Algorithms
    • Dynamic programming, basic data structures, complexity (NP-hardness)
  • Calculus and Linear Algebra
    • positive semi-definiteness, multivariate derivates (be prepared for lots and lots of gradients!)
  • Programming
    • This is a demanding class in terms of programming skills.
    • HWs will involve a mix of languages (Python, C++, and Lua) and libraries (Caffe and Torch).
    • Your language of choice for project.
  • Ability to deal with abstract mathematical concepts


  • The class is full. Can I still get in?

    See the advisors in 340 Whittemore Hall about force-add forms, and come to the first class. There will be a HW0 released on day 1 to check background preparation. Our past experience with HW0 suggests that slots will open up after some students drop the class.

  • I have not taken an “Intro to Machine Learning” class or I am taking it in parallel. Can I still take this class?

    Try working on HW0. If terms seem unfamiliar or you find yourself doing significant background reading to even understand the question, you may benefit from waiting for the next offering of this class.

  • Can I audit this class?

    Depends. We will give preference to students wanting to take this class for credit, so if the class is full, there will be no audits.


  • Length <=1 page
  • 11 pt Times New Roman, 1 inch margins
  • Due: Midnight before class
  • Organization of Review:
  • Summary:
    • What is this paper about?
    • What is the main contribution?
    • Describe the main approach & results. Just facts, no opinions yet.
  • Strengths:
    • Is there a new theoretical insight?
    • Or a significant empirical advance? Did they solve a standing open problem?
    • Or is a good formulation for a new problem?
    • Or a faster/better solution for an existing problem?
    • Any good practical outcome (code, algorithm, etc)?
    • Are the experiments well executed?
    • Useful for the community in general?
  • Weaknesses:
    • What would you do differently?
    • Any missing baselines? missing datasets?
    • any odd design choices in the algorithm not explained well? quality of writing?
    • Is there sufficient novelty in what they propose? Has it already been done? Minor variation of previous work?
    • Why should anyone care? Is the problem interesting and significant?
  • Reflections:
    • How does this relate to other papers we have read?
    • What are the next research directions in this line of work?


  • Once in the semester
  • Expected to:
    • read all papers listed for that day
    • provide overview of that day’s theme
    • present details of at least 2 papers and explain how different papers relate to each other.
    • please clearly cite the source of each slide that is not your own.
    • not submit paper review for that class
    • meet with TA 3-4 days before class to dry run presentation (Worth 40% of presentation grade)


  • You are encouraged to try out interesting applications of deep learning (vision, NLP,computational biology, UAVs, etc!)
  • The project must be done in this semester. No double counting.
  • You may combine with other course project but must delineate the different parts
  • Extra credit for shooting a publication
  • Main Categories:
    • Application/survey - compare a bunch of algorithms on a new application domain of your interest
    • Formulation/Development - Formulate a new model or algorithm for a new/od problem
    • Theory: Theoretically analyze an existing algorithm.

Note to people outside VT

Feel free to use the slides and materials available online here. If you use our slides, an appropriate attribution is requested. Please email the instructor with any corrections or improvements.


We thank NVIDIA for GPU donations and AWS Educate Grant for cloud computing resources.