You’ve probably heard that Deep Learning is making news across the world as one of the most promising techniques in machine learning, especially for analyzing image data. With every industry dedicating resources to unlock the deep learning potential, to be competitive, you will want to use these models in tasks such as image tagging, object recognition, speech recognition, and text analysis.

Introduction to Deep Learning

• Parameter Hyperspace

• Minimizing Cost Entropy

• Normalized Inputs And Initial Weights

• Measuring Performance

• Transition Into Practical Aspects Of Learning

• Stochastic Gradient Descent

• Training your Logistic Classifier

• Transition: Overfitting -> Dataset Size

• Momentum And Learning Rate Decay

• Supervised Classification

• Solving Problems

• Lather Rinse Repeat

• Optimizing A Logistic Classifier

• Cross Entropy

• What is Deep Learning

Deep Neural Network

• "2-layer" neural network

• Network Of ReLUs

• Dropout

• Intro to Deep Neural Network

• No Neurons

• Backprop

• Regularization Intro

• Linear Models Are Limited

• The Chain Rule

• Dropout Pt-2

• Regularization

• Training A Deep Learning Network

Deep Learning Internals

• How They Work

1. A Simple Predicting Machine

2. Following Signals Through A Neural Network

3. Sometimes One Classifier Is Not Enough

4. Learning Weights From More Than One Node

5. A Three Layer Example with Matrix Multiplication

6. Training A Simple Classifier

7. Backpropagating Errors To More Layers

8. Classifying is Not Very Different from Predicting

9. Making it easy by looking at logic and math

10. Preparing Data

11. Matrix Multiplication is Useful Honest!

12. Neurons, Nature’s Computing Machines

13. How Do We Actually Update Weights?

14. Weight Update Worked Example

15. Backpropagating Errors with Matrix Multiplication

16. Backpropagating Errors From More Output Nodes

• DIY with Python

1. Interactive Python = IPython

2. A Very Gentle Start with Python

3. The MNIST Dataset of Handwritten Numbers

4. Python

5. Neural Network with Python

• Hand rolled Neural Network

1. Creating New Training Data: Rotations

2. Your Own Handwriting

3. Inside the Mind of a Neural Network

Software Tools

• Tensorflow

1. Installation

2. Sharing Variables

3. Creating Your First Graph and Running It in a Session

4. Managing Graphs

5. Visualizing the Graph and Training Curves Using TensorBoard

6. Implementing Gradient Descent

7. Lifecycle of a Node Value

8. Linear Regression with TensorFlow

9. Modularity

10. Saving and Restoring Models

11. Name Scopes

12. Feeding Data to the Training Algorithm

• Keras

Artificial Neural Networks Internals

• Training a DNN Using Plain TensorFlow

• Fine-Tuning Neural Network Hyperparameters

• Training an MLP with TensorFlow’s High-Level API

• From Biological to Artificial Neurons

• Deep Feedforward Networks

1. Hidden Units

2. Architecture Design

3. Back-Propagation and Other Differentiation Algorithms

4. Gradient-Based Learning

5. Learning XOR

• Optimization for Training Deep Models

1. Random or Unsupervised Features

2. Structured Outputs

3. Convolutional Networks

4. Challenges in Neural Network Optimizatio

5. Efficient Convolution Algorithms

6. Variants of the Basic Convolution Function

7. The Convolution Operation

8. Pooling

9. Parameter Initialization Strategies

10. Motivation

11. How Learning Differs from Pure Optimization

12. Basic Algorithms

13. Optimization Strategies and Meta-Algorithms

14. The Neuroscientific Basis for Convolutional Networks

15. Convolution and Pooling as an Infinitely Strong Prior

16. Data Types

17. Approximate Second-Order Methods

18. Algorithms with Adaptive Learning Rates

• Regularization for Deep Learning

1. Bagging and Other Ensemble Methods

2. Dataset Augmentation

3. Tangent Distance, Tangent Prop, and Manifold Tangent Classifier

4. Parameter Tying and Parameter Sharing

5. Semi-Supervised Learning

6. Early Stopping

7. Sparse Representations

8. Regularization and Under-Constrained Problems

9. Multi-Task Learning

10. Dropout

11. Norm Penalties as Constrained Optimization

12. Noise Robustness

13. Parameter Norm Penalties

14. Adversarial Training

• Convolutional Neural Networks Internals

1. Pooling layer

2. Image augmentation

3. Convolutional layer

4. History of CNNs

5. Convolutional layers in Keras

6. Code for visualizing an image

7. Input layer

8. How do computers interpret images?

9. Practical example image classification

10. Convolutional neural networks

11. Dropout

• Attention Mechanism for CNN and Visual Models

1. Types of Attention

2. Glimpse Sensor in code

3. Attention mechanism for image captioning

4. Hard Attention

5. Applying the RAM on a noisy MNIST sample

6. Recurrent models of visual attention

7. Using attention to improve visual models

8. Reasons for sub-optimal performance of visual CNN models

9. Soft Attention

• Build Your First CNN and Performance Optimization

1. Convolution and pooling operations in TensorFlow

2. Convolutional operations

3. Using tanh

4. Convolution operations in TensorFlow

5. Regularization

6. Fully connected layer

7. Weight and bias initialization

8. Pooling, stride, and padding operations

9. CNN architectures and drawbacks of DNNs

10. Applying pooling operations in TensorFlow

11. Using sigmoid

12. Training a CNN

13. Using ReLU

14. Activation functions

• Building, training, and evaluating our first CNN

1. Creating a CNN model

2. Defining CNN hyperparameters

3. Model evaluation

4. Dataset description

5. Loading the required packages

6. Running the TensorFlow graph to train the CNN model

7. Preparing the TensorFlow graph

8. Loading the training/test images to generate train/test set

9. Constructing the CNN layers

• Model performance optimization

1. Applying dropout operations with TensorFlow

2. Building the second CNN by putting everything together

3. Appropriate layer placement

4. Which optimizer to use?

5. Creating the CNN model

6. Dataset description and preprocessing

7. Number of neurons per hidden layer

8. Number of hidden layers

9. Batch normalization

10. Memory tuning

11. Training and evaluating the network

12. Advanced regularization and avoiding overfitting

• Popular CNN Model Architectures

1. Architecture insights

2. ResNet architecture

3. AlexNet architecture

4. VGG image classification code example

5. Introduction to ImageNet

6. VGGNet architecture

7. GoogLeNet architecture

8. LeNet

9. Traffic sign classifiers using AlexNet

10. Inception module

• Transfer Learning

1. Multi-task learning

2. Target dataset is small but different from the original training

3. dataset

4. Autoencoders for CNN

5. Applications

6. Target dataset is large and similar to the original training dataset

7. Introducing to autoencoders

8. Convolutional autoencoder

9. Target dataset is large and different from the original training

10. dataset

11. Transfer learning example

12. Feature extraction approach

13. Target dataset is small and is similar to the original training

14. dataset

15. An example of compression

• GAN: Generating New Images with CNN

1. Feature matching

2. GAN code example

3. Deep convolutional GAN

4. Adding the optimizer

5. Training a GAN model

6. Semi-supervised learning and GAN

7. Pixpix - Image-to-Image translation GAN

8. Calculating loss

9. Semi-supervised classification using a GAN example

10. CycleGAN

11. Batch normalization

• Object Detection and Instance Segmentation with CNN

1. Creating the environment

2. Fast R-CNN (fast region-based CNN)

3. The differences between object detection and image classification

4. Mask R-CNN (Instance segmentation with CNN)

5. Cascading classifiers

6. Haar Features

7. Faster R-CNN (faster region proposal network-based CNN)

8. Traditional, nonCNN approaches to object detection

9. R-CNN (Regions with CNN features)

10. Running the pre-trained model on the COCO dataset

11. Why is object detection much more challenging than image

12. classification?

13. The Viola-Jones algorithm

14. Preparing the COCO dataset folder structure

15. Downloading and installing the COCO API and detectron library

16. (OS shell commands)

17. Instance segmentation in code

18. Haar features, cascading classifiers, and the Viola-Jones algorithm

19. Installing Python dependencies (Python environment)

• Popular CNN Model Architectures

1. Introduction to ImageNet

2. VGG image classification code example

3. GoogLeNet architecture

4. Architecture insights

5. Inception module

6. AlexNet architecture

7. VGGNet architecture

8. LeNet

9. ResNet architecture

10. Traffic sign classifiers using AlexNet

Recurrent Neural Network and Sequence Modelling

• Concrete Recurrent Neural Network Architectures

1. Simple RNN

2. Gated Architectures:LSTM

3. Gated Architectures:GRU

4. CBOW as an RNN

5. Dropout in RNNs

6. Gated Architectures:Other Variants

• Recurrent Neural Networks: Modeling Sequences and Stacks

1. Transducer

2. RNN Training

3. RNN Abstraction

4. Common RNN Usage-patterns

5. A Note on Reading the Literature

6. Encoder

7. Multi-layer (stacked) RNNs

8. RNNs for Representing Stacks

9. Acceptor

10. Bidirectional RNNs (biRNN)

• Modeling with Recurrent Networks

1. Acceptors

2. RNN–CNN Document Classification

3. RNNs as Feature Extractors

4. Subject-verb Agreement Grammaticality Detection

5. Arc-factored Dependency Parsing

6. Part-of-speech Tagging

7. Sentiment Classification

• Conditioned Generation

1. Applications

2. Sequence to Sequence Models

3. Syntactic Parsing

4. Morphological Inflection

5. Attention-based Models in NLP

6. Computational Complexity

7. Conditioned Generation with Attention

8. Machine Translation

9. Training Generators

10. Interpretability

11. Other Conditioning Contexts

12. Conditioned Generation (Encoder-Decoder)

13. Unsupervised Sentence Similarity

14. RNN Generators

• Models for Sequence Analysis

1. Dissecting a Neural Translation Network

2. Beam Search and Global Normalization

3. Tackling seqseq with Neural N-Grams

4. Implementing a Sentiment Analysis Model

5. Long Short-Term Memory (LSTM) Units

6. Recurrent Neural Networks

7. Implementing a Part-of-Speech Tagger

8. A Case for Stateful Deep Learning Models

9. Solving seqseq Tasks with Recurrent Neural Networks

10. The Challenges with Vanishing Gradients

11. Augmenting Recurrent Networks with Attention

12. Dependency Parsing and SyntaxNet

13. TensorFlow Primitives for RNN Models

14. Analyzing Variable-Length Inputs

Reinforcement Learning

• Policy Gradient Methods

• Integrating Learning and Planning

• Model-Free Control

• Exploration and Exploitation

• Markov Decision Processes

• Case Study: RL in Classic Games

• Introduction to Reinforcement Learning

• Planning by Dynamic Programming

• Model-Free Prediction

• Value Function Approximation

Deep Generative Models

• Deep Boltzmann Machines

• Back-Propagation through Random Operations

• Restricted Boltzmann Machines

• Boltzmann Machines for Structured or Sequential Outputs

• Generative Stochastic Networks

• Boltzmann Machines

• Other Boltzmann Machines

• Other Generation Schemes

• Directed Generative Nets

• Boltzmann Machines for Real-Valued Data

• Evaluating Generative Models

• Drawing Samples from Autoencoders

• Deep Belief Networks

• Convolutional Boltzmann Machines

Crash Course in GPU

• Introduction to CUDA and OpenCL

• Fundamentals of GPU Algorithms(Applications of Sort and Scan)

• Dynamic Parallelism

• Optimizing GPU Programs

• The GPU Hardware and Parallel Communication Patterns

• Parallel Computation Patterns

• The GPU programming Model

• Parallel Optimization Patterns

• Fundamentals of GPU Algorithms(Reduce,Scan,Histograms)

• Deep Learning use of GPU

Applications

• Other Applications

• Computer Vision

• Natural Language Processing

• Large Scale Deep Learning

• Speech Recognition

Practical Methodology

• Selecting Hyperparameters

• Debugging Strategies

• Example : Facial Recognition

• Performance Metrics

• Determining Whether to Gather More Data

• Default Baseline Models