Dissecting Deep Learning

Deep learning. Pretty much everything you hear about AI in the news today is in some way related to this term. What is deep learning? How does it work? Where can it be used? And how will it be developed in the future?

In this Collection, I’m trying to decompose a massively exploding field into a logical structure to make it comprehensible.

I’ll first cover the general aspects of deep learning, or deep neural networks, if you like. That includes a little bit of history, the components of deep neural networks and how you train them.

I will then dive deeper into various types of neural networks. So far, I’m planning to cover Convolutional Neural Networks, which can be used for image-like data, as well as Recurrent Neural Networks, which are primarily used for timeseries data (although the first can be used as well).

This is followed by a wide array of application types for deep learning models. Finally, based on various sources, I will be trying to construct a set of viewpoints about where people think deep learning is heading to.

I’ve done my utmost best to write in comprehensible English. Since I’m originally not a native speaker, it may unfortunately be the case that I’ve written things down in a weird way or, worse, in a way you don’t understand. In that case, please let me know ASAP so I can correct this. we Dutchies don’t mind directness, so please don’t hesitate and leave a comment below 👇😉

Please note that this is a work in progress. I’m working as hard as I can to get it done ASAP, but it takes time. In order to avoid that I can only publish when it is finished in its entirety (which may or may not happen), I’ve already published what I have so far. If you think it’s interesting, please feel invited to add this page to your bookmarks and check back every now and then. I promise that I’ve done my best to generate new content by then!

Table of contents

Deep neural networks in general

  • What is deep learning?
  • How models learn: Learning Rules.
  • The high-level Deep Learning training process.
  • History of deep neural networks.
  • A 2012 breakthrough in Artificial Intelligence.
  • Deep learning in 2019: hype or revolution?

Weight initialization

Activation functions

Loss and loss functions

  • What is loss?
  • What is a loss function?
  • Which loss function do I need?

Optimizing your model

  • The backpropagation algorithm: history, maths and intuition.
  • Stochastic Gradient Descent.
  • Problems with Stochastic Gradient Descent.
  • Extensions to Stochastic Gradient Descent.
  • What is the Learning Rate?
  • Problems with fixed Learning Rates.
  • Non-fixed Learning Rate: Learning Rate Decay.
  • Non-fixed Learning Rate: Learning Rate Range Test.
  • Non-fixed Learning Rate: Cyclical Learning Rates.

Regularization

  • L1 and L2 Regularization.
  • Batch Normalization.
  • Dropout.

Configuring the Training Process

  • Batch size.
  • Epochs.

Convolutional Neural Networks

Work in progress.

Recurrent Neural Networks

Work in progress.

Application Types for Deep Learning

Work in progress.

The future of deep learning

Work in progress.