Overview of activation functions for neural networks
The neurons of neural networks perform operations that are linear: they multiple an input vector with a weights vector and...
Tag: activation functions
How to use FTSwish with Keras?
Flatten-T Swish is a new (2018) activation function that attempts to find the best of both worlds between traditional ReLU...
What is the FTSwish activation function?
Over the last few years, we’ve seen the rise of a wide range of activation functions – such as FTSwish....
How to use ELU with Keras?
Exponential Linear Unit is a new type of activation function that attempts to resolve both the defects of traditional ReLU...
How to use PReLU with Keras?
Rectified Linear Unit, or ReLU, is considered to be the standard activation function of choice for today’s neural networks. Even...
Beyond Swish: the LiSHT activation function
Deep neural networks perform linear operations to combine weight vectors with input vectors. The values that are the outputs of...
Using Leaky ReLU with Keras
Even though the traditional ReLU activation function is used quite often, it may sometimes not produce a converging model. This...
Leaky ReLU: improving traditional ReLU
The Leaky ReLU is a type of activation function which comes across many machine learning blogs every now and then....
He/Xavier initialization & activation functions: choose wisely
Deep learning models require to be initialized. Their layers have activation functions to make neuron outputs nonlinear. But how to...
Implementing ReLU, Sigmoid and Tanh in Keras
In a recent blog post, we looked at widely used activation functions in today’s neural networks. More specifically, we checked...
