# What are L1, L2 and Elastic Net Regularization in neural networks?

Last Updated on 21 January 2020

When you’re training a neural network, you’re learning a mapping from some input value to a corresponding expected output value. This is great, because it allows you to create predictive models, but who guarantees that the mapping is correct for the data points that aren’t part of your data set?

That is, how do you ensure that your learnt mapping does not oscillate very heavily if you want a smooth function instead?

Regularization can help here. With techniques that take into account the complexity of your weights during optimization, you may steer the networks towards a more general, but scalable mapping, instead of a very data-specific one.

In this blog, we cover these aspects. First, we’ll discuss the need for regularization during model training. We then continue by showing how regularizers can be added to the loss value, and subsequently used in optimization. This is followed by a discussion on the three most widely used regularizers, being L1 regularization (or Lasso), L2 regularization (or Ridge) and L1+L2 regularization (Elastic Net). Finally, we provide a set of questions that may help you decide which regularizer to use in your machine learning project.

Are you ready? Let’s go! 😎

## The need for regularization during model training

When you are training a machine learning model, at a high level, you’re learning a function $$\hat{y}: f(x)$$ which transforms some input value $$x$$ (often a vector, so $$\textbf{x}$$) into some output value $$\hat{y}$$ (often a scalar value, such as a class when classifying and a real number when regressing). \

Contrary to a regular mathematical function, the exact mapping (to $$y$$) is not known in advance, but is learnt based on the input-output mappings present in your training data (so that $$\hat{y} \approx y$$ – hence the name, machine learning 🙂

This understanding brings us to the need for regularization.

### Complex mappings vs simple mappings

Say that you’ve got a dataset that contains points in a 2D space, like this small one:

## Do you want to start learning ML from a developer perspective? 👩‍💻

Blogs at MachineCurve teach Machine Learning for Developers. Sign up to learn new things and better understand concepts you already know. We send emails every Friday.
By signing up, you consent that any information you receive can include services and special offers by email.

## 6 thoughts on “What are L1, L2 and Elastic Net Regularization in neural networks?”

1. Expert

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

1. Chris

Thanks for the compliment!

Best,
Chris