Deep Learning – Simplified

The buzz word is around for a few years in the analytic world, with companies investing heavily to fund the research. From understanding human perception to building self-driven cars deep learning comes with a package of great promises. I was thinking to myself as how I could put these concepts in simple terms which led this blog post. So lets get started.

What is deep learning?

 The foundations of deep learning has it’s inspiration from the ability of the human beings to perceive things as they appear to him. The human perception is  a miracle of nature. Well, lets try answering this – Which of the following pictures has a bi-cycle in it?

dl_first_pic

Well, I quite liked the banana!. But I am sure you would still go for  pictures 2 and 3. It would have taken a second for you to look at the images and make the classification unconsciously. But underlying this decision, we have involved our  entire series of visual cortices (V1 – V5) in the brain each of which contains around 140 million neurons with billions of connections between them forming a network. Yes, we are a living super computer walking on this planet, but fail to recognize the complexity of the problem involved.  Computer scientist who undertook research on computer vision, natural language processing were marveled by this act of the human brain and to help computers mimic the same. This brought in the idea of Artificial Neural Networks (ANN) in the 1980’s

Why not write an algorithm?

 Earlier to deep learning computer scientist wrote machine learning algorithms that take in features from images or text. The major draw back of such approaches in machine learning is that programmers have to constantly tell computers which features they have to use from the raw data. This puts the burden on the programmer to perform feature engineering and no wonder we see that the algorithms are not performing well. It also becomes increasingly difficult for traditional machine algorithms to work in case of learning complex patterns.

Unlike the traditional machine learning algorithms, Deep learning (based on ANN) has gone past this barrier as it uses training examples for the system to learn the features on its own.

What!! But how?

To get started on neural networks work it is good to understand how a artificial neuron called a perceptron functions.  A Perceptron (schematic representation below) takes in inputs (Xi) and has specific weights (Wi) which assign importance to each of these inputs. The neuron computes a function based on the weighted inputs.

ANN_Percentron

For the sake of simplicity, let us assume that the computed function is linear in nature.  If the output of the computed function exceeds a specific threshold or bias the neurons outputs a 1 or 0. Lets look at the example below.

Let us assume we are charted on a weekly exercise plan to burn 1400 calories (threshold) per week. Let us assume the inputs (walking, running, swimming) x1,x2,x3 each burn  calories 100,150,200 respectively. It is for us to decide to plan the weekly schedule of activities to maintain a healthy living. The number of days (weights) we choose to exercise is up to us. Let us model this problem with a linear perceptron. As you could note by choosing the weights (number of days we do an activity) and threshold (calories to burn this week) we could decide the output (healthy week)

ann_example

While the idea of a perceptron is only an inspiration from the human brain, we are still far from understanding and replicating on how it works. A neural network is formed by hooking these neurons as shown below to each other and help making complex decisions.  The structure of a neural network could be divided in to 3 layers as shown below. The left most layer are called input layer which contains the input neurons, the right most layer contains is called the output layer containing the output neurons. Unlike the figure below each of these layers can have multiple nodes. The inner layer of neurons are are called the hidden layer. The first vanilla version of artificial neural networks are called Multi Layer Perceptrons (MLP).

neural_network

The Multi Layer Perceptrons (MLP) is a type of supervised network as it requires an label or desired output to learn. A Multilayer Perceptron (MLP) is a type of neural network referred to as a supervised network because it requires a desired output in order to learn.  As reach in machine layer matured each of these nodes were replaced by more sophisticated algorithms than the perceptrons. The neural network have to initially trained to learn the abstract model. In a neural network the output of one layer are constantly fed forward as inputs to the next layer. This approach is called feed-forward propagation.

 

Each of these nodes learn a function and feed their output forward to the next layer in the network. Not every time a node picks a function to be learned correctly. The difference between the expected output and the current output of a node are called errors (shown below).  These errors prevent the neural networks from learning effectively. Research and usage of neural network seemed to have met a trough until the advent of back propagation algorithm.

backprop

The back-propagation algorithm acts as an error correcting mechanism at each neuron level, there by helping the network to learn effectively. The derivation of how backward propagation helps to solve the problem is beyond the scope of this post.

 Deep Neural Network

A deep neural network (DNN) is an artificial neural network (ANN) with multiple hidden layers of units between the input and output layers. Each of the node learn an effective function and transfer the knowledge forward.  Let us consider the working of one of the deep network architectures used for image classification. Pixels from the input image are converted to feature maps (smaller or abstract representation). These abstract representation are then used  to form simpler forms by the next layer and finally in a more generic representation.

feature_map_layers

Let us take a look at an example of image classification of human faces as shown below. The input layers learn to recognize lines or edges of the face (initial feature map). This information is fed forward to layer of nodes in the hidden layer which then again learns a more abstract concept like the eyes, nose. The final layer then assimilates this learning to form the object model of the face.

learning_by_nn

Research[4] in the topic of deep learning is constantly growing and there are constant updates to architectures and algorithms used in deep neural network. Deep learning has been extensively applied in the fields of image recognition, Natural Language processing, Audio recognition and information retrieval.Details of all the architectures and algorithms are beyond the scope of this post and would be handled in a different post.

Hype or Real?

The concept of neural networks to learn have actually existed for decades, but there were major problems building larger networks. Deep learning is partly hyped or could be a re-branded version of neural network. But the recent hype about deep learning and neural network is because of the ability to perform better than the kernel algorithms on standard data sets. Thanks to high performance computation units like GPU and better parallel algorithms neural network have grown bigger and deeper.

References

  1. Neural networks and deep learning
  2. Deep learning – Wikipedia, the free encyclopedia
  3. Deng, Li et el;Tutorial Survey of Architectures, algorithms and applications of deep learning
  4. Deep Learning, Self-Taught Learning and Unsupervised Feature Learning