Deep learning. Pretty much everything you hear about AI in the news today is in some way related to this term. What is deep learning? How does it work? Where can it be used? And how will it be developed in the future?
In this Collection, I’m trying to decompose a massively exploding field into a logical structure to make it comprehensible.
I’ll first cover the general aspects of deep learning, or deep neural networks, if you like. That includes a little bit of history, the components of deep neural networks and how you train them.
I will then dive deeper into various types of neural networks. So far, I’m planning to cover Convolutional Neural Networks, which can be used for image-like data, as well as Recurrent Neural Networks, which are primarily used for timeseries data (although the first can be used as well).
This is followed by a wide array of application types for deep learning models. Finally, based on various sources, I will be trying to construct a set of viewpoints about where people think deep learning is heading to.
I’ve done my utmost best to write in comprehensible English. Since I’m originally not a native speaker, it may unfortunately be the case that I’ve written things down in a weird way or, worse, in a way you don’t understand. In that case, please let me know ASAP so I can correct this. we Dutchies don’t mind directness, so please don’t hesitate and leave a comment below 👇😉
Please note that this is a work in progress. I’m working as hard as I can to get it done ASAP, but it takes time. In order to avoid that I can only publish when it is finished in its entirety (which may or may not happen), I’ve already published what I have so far. If you think it’s interesting, please feel invited to add this page to your bookmarks and check back every now and then. I promise that I’ve done my best to generate new content by then!
Table of contents
Deep neural networks in general
- What is deep learning?
- Machine Learning Error: Bias, Variance and Irreducible Error with Python
- How models learn: Learning Rules.
- The high-level Deep Learning training process.
- History of deep neural networks.
- A 2012 breakthrough in Artificial Intelligence.
- Deep learning in 2019: hype or revolution?
Weight initialization
- What is weight initialization?
- Random initialization: vanishing and exploding gradients.
- He and Xavier initialization vs activation functions.
Activation functions
- What is an activation function?
- Overview of activation functions for neural networks
- Why could a linear activation function be useless?
- Why nonlinear activation functions improve ML performance – with TensorFlow example
- ReLU, Sigmoid and Tanh: today’s most used activation functions.
- Leaky ReLU: improving traditional ReLU.
- Best of both worlds: the Swish activation function.
- Beyond Swish: the LiSHT activation function
- What is the FTSwish activation function?
- How does the Softmax activation function work?
Loss and loss functions
- What is loss – and what are loss functions?
- Which loss function do I need?
Optimizing your model
- The backpropagation algorithm: history, maths and intuition.
- Gradient Descent and its variants
- Extensions to Gradient Descent: adaptive optimizers
- What is the Learning Rate?
- Problems with fixed and decaying Learning Rates.
- Non-fixed Learning Rate: Learning Rate Decay.
- Non-fixed Learning Rate: Learning Rate Range Test.
- Non-fixed Learning Rate: Cyclical Learning Rates.
- Escaping Saddle Points / Local Minima with Adjusted Learning Rates
- How to check if your Deep Learning model is underfitting or overfitting?
Regularization and Normalization
Configuring the Training Process
- Batch size.
- Epochs.
Convolutional Neural Networks
- Convolutional Neural Networks and their components for computer vision;
- Understanding separable convolutions.
- Understanding transposed convolutions.
- What are Max Pooling, Average Pooling, Global Max Pooling and Global Average Pooling?
- What is padding in a neural network?
Recurrent Neural Networks
- From vanilla RNNs to Transformers: a history of Seq2Seq learning
- An introduction to Long Short-Term Memory networks (LSTM)
Other architectures
Transformers
- From vanilla RNNs to Transformers: a history of Seq2Seq learning
- Dissecting the Transformer architecture
Generative models
Application Types for Deep Learning
The future of deep learning
Work in progress.
