Last Updated on 6 January 2021

Pure signals only exist in theory. That is, when you’re doing signal processing related activities, it’s very likely that you’ll experience noise. Whether noise is caused by the measurement (or reception) device or by the medium in which you perform measurements, you want it gone.

Various mathematical tricks exist to filter out noise from a signal. When noise is relatively constant across a range of signals, for example, you can take the mean of all the signals and deduct it from each individual signal – which likely removes the factors that contribute from noise.

However, these tricks work by knowing a few things about the noise up front. In many cases, the exact shape of your noise is unknown or cannot be estimated because it is relatively hidden. In those cases, the solution may lie in *learning* the noise from example data.

Autoencoders can be used for this purpose. By feeding them noisy data as inputs and clean data as outputs, it’s possible to make them recognize *the ideosyncratic noise for the training data*. This way, autoencoders can serve as denoisers.

But what are autoencoders exactly? And why does the way they work make them suitable for noise removal? And how to implement one for *signal denoising / noise reduction*?

We’ll answer these questions in today’s blog. First, we’ll provide a recap on autoencoders – to (re)gain a theoretical understanding of what they are and how they work. This includes a discussion on why they can be applied to noise removal. Subsequently, we implement an autoencoder to demonstrate this, by means of a three-step process:

- We generate a large dataset of \(x^2\) samples.
- We generate a large dataset of \(x^2\) samples to which Gaussian (i.e., random) noise has been added.
- We create an autoencoder which learns to transform noisy \(x^2\) inputs into the original sine, i.e.
*which removes the noise*– also for new data!

Ready?

Okay, let’s go! 😊

**Update 06/Jan/2021:** updated the article to reflect TensorFlow in 2021. As 1-dimensional transposed convolutions are available in TensorFlow now, the article was updated to use `Conv1D`

and `Conv1DTranspose`

layers instead of their 2D variants. This fits better given the 1D aspect of our dataset. In addition, references to old Keras were replaced with newer `tf.keras`

versions, meaning that this article is compatible with TensorFlow 2.4.0+.

## Table of contents

## Recap: what are autoencoders?

If we’re going to build an autoencoder, we must know what they are.

In our blog post **“Conv2DTranspose: using 2D transposed convolutions with Keras”**, we already covered the high-level principles behind autoencoders, but it’s wise to repeat them here.

We can visualize the flow of an autoencoder as follows:

Autoencoders are composed of two parts: an *encoder*, which encodes some input into encoded state, and a *decoder* which can decode the encoded state into another format. This can be a reconstruction of the original input, as we can see in the plot below, but it can also be something different.

For example, autoencoders are learnt for noise removal, but also for dimensionality reduction (Keras Blog , n.d.; we then use them to convert the input data into low-dimensional format, which might benefit training lower-dimensionality model types such as SVMs).

## Let's pause for a second! 👩💻

**free**Machine Learning update today! You will learn

**new things**and

**better understand**concepts you already know.

We send emails at least every Friday. Welcome!

Note that the red parts in the block above – that is, the encoder and the decoder, are *learnt based on data* (Keras Blog, n.d.). This means that, contrary to more abstract mathematical functions (e.g. filters), they are highly specialized in *one domain* (e.g. signal noise removal at \(x^2\) plots as we will do next) while they perform very poorly in another (e.g. when using the same autoencoder for image noise removal).

## Why autoencoders are applicable to noise removal

Autoencoders learn an *encoded state* with an *encoder*, and learn to decode this state into *something else* with a *decoder*.

Now think about this in the context of signal noise: suppose that you feed the neural network noisy data as *features*, while you have the pure data available as *targets*. Following the drawing above, the neural network will learn an encoded state based on the noisy image, and will attempt to decode it to best match the *pure data*. What’s the thing that stands in between the pure data and the noisy data? Indeed, the noise. In effect, the autoencoder will thus learn to recognize noise and remove it from the input image.

Let’s now see if we can create such an autoencoder with Keras.

## Today’s example: a Keras based autoencoder for noise removal

In the next part, we’ll show you how to use the Keras deep learning framework for creating a *denoising* or *signal removal* autoencoder. Here, we’ll first take a look at two things – the data we’re using as well as a high-level description of the model.

### The data

First, the data. As *pure signals* (and hence autoencoder targets), we’re using pure \(x^2\) samples from a small domain. When plotted, a sample looks like this:

For today’s model, we use 100.000 samples. To each of them, we add Gaussian noise – or random noise. While the global shape remains present, it’s clear that the plots become noisy:

### The model

Now, the model. It looks as follows:

…and has these layers:

- The input layer, which takes the input data;
- Two Conv1D layers, which serve as
*encoder*; - Two Conv1D transpose layers, which serve as
*decoder*; - One Conv1D layer with one output, a Sigmoid activation function and padding, serving as the output layer.

To provide more details, this is the model summary:

```
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv1d (Conv1D) (None, 148, 128) 512
_________________________________________________________________
conv1d_1 (Conv1D) (None, 146, 32) 12320
_________________________________________________________________
conv1d_transpose (Conv1DTran (None, 148, 32) 3104
_________________________________________________________________
conv1d_transpose_1 (Conv1DTr (None, 150, 128) 12416
_________________________________________________________________
conv1d_2 (Conv1D) (None, 150, 1) 385
=================================================================
Total params: 28,737
Trainable params: 28,737
Non-trainable params: 0
```

Code language: PHP (php)

Let’s now start with the first part – generating the pure waveforms! Open up your Explorer, navigate to some folder (e.g. `keras-autoencoders`

) and create a file called `signal_generator.py`

. Next, open this file in your code editor – and let the coding process begin!

## Generating pure waveforms

Generating pure waveforms consists of the following steps, in order to generate visualizations like the one shown on the right:

- Adding the necessary imports to the start of your Python script;
- Defining configuration settings for the signal generator;
- Generating the data, a.k.a. the pure waveforms;
- Saving the waveforms and visualizing a subset of them.

### Adding imports

First, the imports – it’s a simple list:

```
import matplotlib.pyplot as plt
import numpy as np
```

Code language: JavaScript (javascript)

We use Numpy for data generation & processing and Matplotlib for visualizing some of the samples at the end.

### Configuring the generator

Generator configuration consists of three steps: sample-wide configuration, intra-sample configuration and other settings. First, sample-wide configuration, which is just the number of samples to generate:

## Never miss new Machine Learning articles ✅

**free**Machine Learning update today! You will learn

**new things**and

**better understand**concepts you already know.

We send emails at least every Friday. Welcome!

```
# Sample configuration
num_samples = 100000
```

Code language: PHP (php)

Followed by intra-sample configuration:

```
# Intrasample configuration
num_elements = 1
interval_per_element = 0.01
total_num_elements = int(num_elements / interval_per_element)
starting_point = int(0 - 0.5*total_num_elements)
```

Code language: PHP (php)

`num_elements`

represents the *width* of your domain. `interval_per_element`

represents the step size that the iterator will take when generating the sample. In this case, the domain \((0, 1]\) will thus contain 100 samples (as \(1/interval per element = 1/0.01 = 100\)). That’s what’s represented in `total_num_elements`

.

The starting point determines where to start the generation process.

Finally, you can set the number of samples that you want visualized in the `other configuration`

settings:

```
# Other configuration
num_samples_visualize = 1
```

Code language: PHP (php)

### Generating data

Next step, creating some data! 😁

We’ll first specify the lists that contain our data and the sub-sample data (one sample in `samples`

contains multiple `xs`

and `ys`

; when \(totalnumelements = 100\), that will be 100 of them each):

```
# Containers for samples and subsamples
samples = []
xs = []
ys = []
```

Code language: PHP (php)

Next, the actual data generation part:

```
# Generate samples
for j in range(0, num_samples):
# Report progress
if j % 100 == 0:
print(j)
# Generate wave
for i in range(starting_point, total_num_elements):
x_val = i * interval_per_element
y_val = x_val * x_val
xs.append(x_val)
ys.append(y_val)
# Append wave to samples
samples.append((xs, ys))
# Clear subsample containers for next sample
xs = []
ys = []
```

Code language: PHP (php)

We’ll first iterate over every sample, determined by the range between 0 and the `num_samples`

variable. This includes a progress report every 100 samples.

Next, we construct the wave step by step, adding the function outputs to the `xs`

and `ys`

variables.

Subsequently, we append the entire wave to the `samples`

list, and clear the subsample containers for generating the next sample.

### Saving & visualizing

The next step is to save the data. We do so by using Numpy’s `save`

call, and save `samples`

to a file called `./signal_waves_medium.py`

.

```
# Input shape
print(np.shape(np.array(samples[0][0])))
# Save data to file for re-use
np.save('./signal_waves_medium.npy', samples)
# Visualize a few random samples
for i in range(0, num_samples_visualize):
random_index = np.random.randint(0, len(samples)-1)
x_axis, y_axis = samples[random_index]
plt.plot(x_axis, y_axis)
plt.title(f'Visualization of sample {random_index} ---- y: f(x) = x^2')
plt.show()
```

Code language: PHP (php)

Next, with some basic Matplotlib code, we visualize `num_samples_visualize`

random samples from the `samples`

array. And that’s it already!

Run your code with `python signal_generator.py`

(ensure that you have Numpy and Matplotlib installed) and the generation process should begin, culminating in a `.npy`

file and one (or more) visualizations popping up once the process finishes.

### Full generator code

If you wish to obtain the entire signal generator at once, here you go:

```
import matplotlib.pyplot as plt
import numpy as np
# Sample configuration
num_samples = 100000
# Intrasample configuration
num_elements = 1
interval_per_element = 0.01
total_num_elements = int(num_elements / interval_per_element)
starting_point = int(0 - 0.5*total_num_elements)
# Other configuration
num_samples_visualize = 1
# Containers for samples and subsamples
samples = []
xs = []
ys = []
# Generate samples
for j in range(0, num_samples):
# Report progress
if j % 100 == 0:
print(j)
# Generate wave
for i in range(starting_point, total_num_elements):
x_val = i * interval_per_element
y_val = x_val * x_val
xs.append(x_val)
ys.append(y_val)
# Append wave to samples
samples.append((xs, ys))
# Clear subsample containers for next sample
xs = []
ys = []
# Input shape
print(np.shape(np.array(samples[0][0])))
# Save data to file for re-use
np.save('./signal_waves_medium.npy', samples)
# Visualize a few random samples
for i in range(0, num_samples_visualize):
random_index = np.random.randint(0, len(samples)-1)
x_axis, y_axis = samples[random_index]
plt.plot(x_axis, y_axis)
plt.title(f'Visualization of sample {random_index} ---- y: f(x) = x^2')
plt.show()
```

Code language: PHP (php)

## Adding noise to pure waveforms

The second part: adding noise to the 100k pure waveforms we generated in the previous step.

## Join hundreds of other learners! 😎

**free**Machine Learning update today! You will learn

**new things**and

**better understand**concepts you already know.

We send emails at least every Friday. Welcome!

It’s composed of these individual steps:

- Once again, adding imports;
- Setting the configuration variables for the noising process;
- Loading the data;
- Adding the noise;
- Saving the noisy samples and visualizing a few of them.

Create an additional file, e.g. `signal_apply_noise.py`

, and let’s add the following things.

### Adding imports

Our imports are the same as we used in the signal generator:

```
import matplotlib.pyplot as plt
import numpy as np
```

Code language: JavaScript (javascript)

### Configuring the noising process

Our noising configuration is also a lot simpler:

```
# Sample configuration
num_samples_visualize = 1
noise_factor = 0.05
```

Code language: PHP (php)

`num_samples_visualize`

is the number of samples we wish to visualize once the noising process finishes, and `noise_factor`

is the amount of noise we’ll be adding to our samples (\(0 = no noise; 1 = full noise\)).

### Loading data

Next, we load the data and assign the samples to the correct variables, being `x_val`

and `y_val`

.

```
# Load data
data = np.load('./signal_waves_medium.npy')
x_val, y_val = data[:,0], data[:,1]
```

Code language: PHP (php)

### Adding noise

Next, we add the noise to our samples.

```
# Add noise to data
noisy_samples = []
for i in range(0, len(x_val)):
if i % 100 == 0:
print(i)
pure = np.array(y_val[i])
noise = np.random.normal(0, 1, pure.shape)
signal = pure + noise_factor * noise
noisy_samples.append([x_val[i], signal])
```

Code language: PHP (php)

First, we define a new list that will contain our noisy samples. Subsequently, we iterate over each sample (reporting progress every 100 samples). We then do a couple of things:

- We assign the pure sample (i.e., the \(x^2\) plot wihtout noise) to the
`pure`

variable. - Subsequently, we generate Gaussian noise using
`np.random.normal`

, with the same shape as`pure`

‘s. - Next, we add the noise to the pure sample, using the
`noise_factor`

. - Finally, we append the sample’s domain and the noisy sample to the
`noisy_samples`

array.

### Saving & visualizing

Next, we – and this is no different than with the generator before – save the data into a `.npy`

file (this time, with a different name 😃) and visualize a few random samples based on the number you configured earlier.

```
# Save data to file for re-use
np.save('./signal_waves_noisy_medium.npy', noisy_samples)
# Visualize a few random samples
for i in range(0, num_samples_visualize):
random_index = np.random.randint(0, len(noisy_samples)-1)
x_axis, y_axis = noisy_samples[random_index]
plt.plot(x_axis, y_axis)
plt.title(f'Visualization of noisy sample {random_index} ---- y: f(x) = x^2')
plt.show()
```

Code language: PHP (php)

If you would now run `signal_apply_noise.py`

, you’d get 100k noisy samples, with which we can train the autoencoder we’ll build next.

### Full noising code

If you’re interested in the full code of the noising script, here you go:

```
import matplotlib.pyplot as plt
import numpy as np
# Sample configuration
num_samples_visualize = 1
noise_factor = 0.05
# Load data
data = np.load('./signal_waves_medium.npy')
x_val, y_val = data[:,0], data[:,1]
# Add noise to data
noisy_samples = []
for i in range(0, len(x_val)):
if i % 100 == 0:
print(i)
pure = np.array(y_val[i])
noise = np.random.normal(0, 1, pure.shape)
signal = pure + noise_factor * noise
noisy_samples.append([x_val[i], signal])
# Save data to file for re-use
np.save('./signal_waves_noisy_medium.npy', noisy_samples)
# Visualize a few random samples
for i in range(0, num_samples_visualize):
random_index = np.random.randint(0, len(noisy_samples)-1)
x_axis, y_axis = noisy_samples[random_index]
plt.plot(x_axis, y_axis)
plt.title(f'Visualization of noisy sample {random_index} ---- y: f(x) = x^2')
plt.show()
```

Code language: PHP (php)

## Creating the autoencoder

It’s now time for the interesting stuff: creating the autoencoder 🤗

Creating it contains these steps:

- Once again, adding some imports 😋
- Setting configuration details for the model;
- Data loading and preparation;
- Defining the model architecture;
- Compiling the model and starting training;
- Visualizing denoised waveforms from the test set, to find out visually whether it works.

To run it successfully, you’ll need **TensorFlow 2.4.0+**, **Matplotlib** and **Numpy**.

Let’s create a third (and final 😋) file: `python signal_autoencoder.py`

.

## We help you with Machine Learning! 🧠

**free**Machine Learning update today! You will learn

**new things**and

**better understand**concepts you already know.

We send emails at least every Friday. Welcome!

### Adding imports

First, let’s specify the imports:

```
import tensorflow.keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv1D, Conv1DTranspose
from tensorflow.keras.constraints import max_norm
import matplotlib.pyplot as plt
import numpy as np
import math
```

Code language: JavaScript (javascript)

From Keras, we import the Sequential API (which we use to stack the layers on top of each other), the Conv1D and Conv1DTranspose layers (see the architecture and the rationale here to find out why), and the MaxNorm constraint, in order to keep the weight updates in check. We also import Matplotlib, Numpy and the Python `math`

library.

### Model configuration

Next, we set some configuration options for the model:

```
# Model configuration
input_shape = (150, 1)
batch_size = 150
no_epochs = 5
train_test_split = 0.3
validation_split = 0.2
verbosity = 1
max_norm_value = 2.0
```

Code language: PHP (php)

Here are some insights about the model configuration:

- The
`input_shape`

, in line with Conv1D input, is thus \( (150, 1)\). - The batch size is 150. This number seemed to work well, offering a nice balance between loss value and prediction time.
- The number of epochs is fairly low, but pragmatic: the autoencoder did not improve substantially anymore after this number.
- We use 30% of the total data, i.e. 30k samples, as testing data.
- 20% of the training data (70k) will be used for validation purposes. Hence, 14k will be used to validate the model per epoch (and even per minibatch), while 56k will be used for actual training.
- All model outputs are displayed on screen, with
`verbosity`

mode set to True. - The
`max_norm_value`

is 2.0. This value worked well in a different scenario, and slightly improved the training results.

### Data loading & preparation

The next thing to do is to load the data. We load both the noisy and the pure samples into their respective variables:

```
# Load data
data_noisy = np.load('./signal_waves_noisy_medium.npy')
x_val_noisy, y_val_noisy = data_noisy[:,0], data_noisy[:,1]
data_pure = np.load('./signal_waves_medium.npy')
x_val_pure, y_val_pure = data_pure[:,0], data_pure[:,1]
```

Code language: PHP (php)

Next, we’ll reshape the data. We do so for each sample. This includes the following steps:

- First, given the way how binary crossentropy loss works, we normalize our samples to fall in the range \([0, 1]\). Without this normalization step, odd loss values (extremely negative ones, impossible with BCE loss) start popping up (Quetzalcohuatl, n.d.).
- We subsequently add the noisy and pure samples to the specific
`*_r`

arrays.

```
# Reshape data
y_val_noisy_r = []
y_val_pure_r = []
for i in range(0, len(y_val_noisy)):
noisy_sample = y_val_noisy[i]
pure_sample = y_val_pure[i]
noisy_sample = (noisy_sample - np.min(noisy_sample)) / (np.max(noisy_sample) - np.min(noisy_sample))
pure_sample = (pure_sample - np.min(pure_sample)) / (np.max(pure_sample) - np.min(pure_sample))
y_val_noisy_r.append(noisy_sample)
y_val_pure_r.append(pure_sample)
y_val_noisy_r = np.array(y_val_noisy_r)
y_val_pure_r = np.array(y_val_pure_r)
noisy_input = y_val_noisy_r.reshape((y_val_noisy_r.shape[0], y_val_noisy_r.shape[1], 1))
pure_input = y_val_pure_r.reshape((y_val_pure_r.shape[0], y_val_pure_r.shape[1], 1))
```

Code language: PHP (php)

Once each sample is resampled, we convert the *entire* array for both the resampled noisy and resampled pure samples into a structure that TensorFlow/Keras can handle. That is, we increase the shape with another dimension to represent the number of channels, which in our case is just 1.

Finally, we perform the split into training and testing data (30k test, 56+14 = 70k train):

```
# Train/test split
percentage_training = math.floor((1 - train_test_split) * len(noisy_input))
noisy_input, noisy_input_test = noisy_input[:percentage_training], noisy_input[percentage_training:]
pure_input, pure_input_test = pure_input[:percentage_training], pure_input[percentage_training:]
```

Code language: PHP (php)

### Creating the model architecture

This is the architecture of our autoencoder:

```
# Create the model
model = Sequential()
model.add(Conv1D(128, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform', input_shape=input_shape))
model.add(Conv1D(32, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv1DTranspose(32, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv1DTranspose(128, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv1D(1, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='sigmoid', padding='same'))
model.summary()
```

Code language: PHP (php)

- We’ll use the Sequential API, for stacking the layers on top of each other.
- The two Conv1D layers serve as the
*encoder*, and learn 128 and 32 filters, respectively. They activate with the ReLU activation function, and by consequence require He initialization. Max-norm regularization is applied to each of them. - The two Conv1DTranspose layers, which learn 32 and 128 filters, serve as the
*decoder*. They also use ReLU activation and He initialization, as well as Max-norm regularization. - The final Conv layer serves as the output layer, and does (by virtue of
`padding='same'`

) not alter the shape, except for the number of channels (back into 1). - Kernel sizes are 3 pixels.

Generating a model summary, i.e. calling `model.summary()`

, results in this summary, which also shows the number of parameters that is trained:

```
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv1d (Conv1D) (None, 148, 128) 512
_________________________________________________________________
conv1d_1 (Conv1D) (None, 146, 32) 12320
_________________________________________________________________
conv1d_transpose (Conv1DTran (None, 148, 32) 3104
_________________________________________________________________
conv1d_transpose_1 (Conv1DTr (None, 150, 128) 12416
_________________________________________________________________
conv1d_2 (Conv1D) (None, 150, 1) 385
=================================================================
Total params: 28,737
Trainable params: 28,737
Non-trainable params: 0
_________________________________________________________________
```

Code language: PHP (php)

### Model compilation & starting the training process

The next thing to do is to compile the model (i.e., specify the optimizer and loss function) and to start the training process. We use Adam and Binary crossentropy for the fact that they are relatively default choices for today’s deep learning models.

Fitting the data shows that we’re going from `noisy_input`

(features) to `pure_input`

(targets). The number of epochs, the batch size and the validation split are as configured earlier.

```
# Compile and fit data
model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(noisy_input, pure_input,
epochs=no_epochs,
batch_size=batch_size,
validation_split=validation_split)
```

Code language: PHP (php)

### Visualizing denoised waveforms from test set

Once the training process finishes, it’s time to find out whether our model actually works. We do so by generating a few reconstructions: we add a noisy sample from the test set (which is data the model has never seen before!) and visualize whether it outputs the noise-free shape. This is the code

```
# Generate reconstructions
num_reconstructions = 4
samples = noisy_input_test[:num_reconstructions]
reconstructions = model.predict(samples)
# Plot reconstructions
for i in np.arange(0, num_reconstructions):
# Prediction index
prediction_index = i + percentage_training
# Get the sample and the reconstruction
original = y_val_noisy[prediction_index]
pure = y_val_pure[prediction_index]
reconstruction = np.array(reconstructions[i])
# Matplotlib preparations
fig, axes = plt.subplots(1, 3)
# Plot sample and reconstruciton
axes[0].plot(original)
axes[0].set_title('Noisy waveform')
axes[1].plot(pure)
axes[1].set_title('Pure waveform')
axes[2].plot(reconstruction)
axes[2].set_title('Conv Autoencoder Denoised waveform')
plt.show()
```

Code language: PHP (php)

Open up your terminal again, and run `python signal_autoencoder.py`

. Now, the training process should begin.

### Full model code

If you’re interested in the full code, here you go:

```
import tensorflow.keras
from tensorflow.keras.models import Sequential, save_model
from tensorflow.keras.layers import Conv1D, Conv1DTranspose
from tensorflow.keras.constraints import max_norm
import matplotlib.pyplot as plt
import numpy as np
import math
# Model configuration
input_shape = (150, 1)
batch_size = 150
no_epochs = 5
train_test_split = 0.3
validation_split = 0.2
verbosity = 1
max_norm_value = 2.0
# Load data
data_noisy = np.load('./signal_waves_noisy_medium.npy')
x_val_noisy, y_val_noisy = data_noisy[:,0], data_noisy[:,1]
data_pure = np.load('./signal_waves_medium.npy')
x_val_pure, y_val_pure = data_pure[:,0], data_pure[:,1]
# Reshape data
y_val_noisy_r = []
y_val_pure_r = []
for i in range(0, len(y_val_noisy)):
noisy_sample = y_val_noisy[i]
pure_sample = y_val_pure[i]
noisy_sample = (noisy_sample - np.min(noisy_sample)) / (np.max(noisy_sample) - np.min(noisy_sample))
pure_sample = (pure_sample - np.min(pure_sample)) / (np.max(pure_sample) - np.min(pure_sample))
y_val_noisy_r.append(noisy_sample)
y_val_pure_r.append(pure_sample)
y_val_noisy_r = np.array(y_val_noisy_r)
y_val_pure_r = np.array(y_val_pure_r)
noisy_input = y_val_noisy_r.reshape((y_val_noisy_r.shape[0], y_val_noisy_r.shape[1], 1))
pure_input = y_val_pure_r.reshape((y_val_pure_r.shape[0], y_val_pure_r.shape[1], 1))
# Train/test split
percentage_training = math.floor((1 - train_test_split) * len(noisy_input))
noisy_input, noisy_input_test = noisy_input[:percentage_training], noisy_input[percentage_training:]
pure_input, pure_input_test = pure_input[:percentage_training], pure_input[percentage_training:]
# Create the model
model = Sequential()
model.add(Conv1D(128, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform', input_shape=input_shape))
model.add(Conv1D(32, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv1DTranspose(32, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv1DTranspose(128, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv1D(1, kernel_size=3, kernel_constraint=max_norm(max_norm_value), activation='sigmoid', padding='same'))
model.summary()
# Compile and fit data
model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(noisy_input, pure_input,
epochs=no_epochs,
batch_size=batch_size,
validation_split=validation_split)
# Generate reconstructions
num_reconstructions = 4
samples = noisy_input_test[:num_reconstructions]
reconstructions = model.predict(samples)
# Plot reconstructions
for i in np.arange(0, num_reconstructions):
# Prediction index
prediction_index = i + percentage_training
# Get the sample and the reconstruction
original = y_val_noisy[prediction_index]
pure = y_val_pure[prediction_index]
reconstruction = np.array(reconstructions[i])
# Matplotlib preparations
fig, axes = plt.subplots(1, 3)
# Plot sample and reconstruciton
axes[0].plot(original)
axes[0].set_title('Noisy waveform')
axes[1].plot(pure)
axes[1].set_title('Pure waveform')
axes[2].plot(reconstruction)
axes[2].set_title('Conv Autoencoder Denoised waveform')
plt.show()
```

Code language: PHP (php)

## Results

Next, the results 😎

After the fifth epoch, validation loss \(\approx 0.3556\). This is high, but acceptable. What’s more important is to find out how well the model works when visualizing the test set predictions.

Here they are:

Clearly, the autoencoder has learnt to remove much of the noise. As you can see, the denoised samples are not entirely noise-free, but it’s a lot better. Some nice results! 😎

## Summary

In this blog post, we created a denoising / noise removal autoencoder with Keras, specifically focused on signal processing. By generating 100.000 pure and noisy samples, we found that it’s possible to create a trained noise removal algorithm that is capable of removing specific noise from input data. I hope you’ve learnt something today, and if you have any questions or remarks – please feel free to leave a comment in the comments box below! 😊 I’ll try to answer your comment as soon as I can.

Thanks for reading MachineCurve today and happy engineering! 😎

*Please note that the code for these models is also available in my keras-autoencoders Github repository.*

## References

Keras Blog. (n.d.). Building Autoencoders in Keras. Retrieved from https://blog.keras.io/building-autoencoders-in-keras.html

Quetzalcohuatl. (n.d.). The loss becomes negative · Issue #1917 · keras-team/keras. Retrieved from https://github.com/keras-team/keras/issues/1917#issuecomment-451534575

TensorFlow. (n.d.). *Tf.keras.layers.Conv1DTranspose*. https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1DTranspose

TensorFlow. (n.d.). *Tf.keras.layers.Conv1D*. https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D

Thank you Chris! Very much appreciate the tutorial

Hi Alex,

Thank you for your compliment 🙂

Regards,

Chris

Hi Chris,

Do you have any idea on how to add temporality to the model?

Hi Marcia,

It’s possible to create signal denoising autoencoders with e.g. LSTM layers, which could allow you to handle time series.

Do you perhaps have a little bit more context?

Regards,

Chris

How did you ensure, that the model did not learn the x^2 signal?

In other words: shouldn’t we test the trained model with other signal forms too?

Hi Ernst,

That’s a fair question, because the noise removal model in this article was trained with samples from x^2 only. In situations like these, neural networks tend to generalize poorly outside situations they haven’t encountered in their training data. To find out more about your assumption, I retrained the model on x^2, which goes smoothly:

I then generated new samples using sin(x), visualized them on the same domain, as well as the denoised outcome:

Clearly, the model also captures part of the function x^2.

To fix this in your model, I would suggest to expand your training set with the signals that can be present in your situation, then let the model figure out that the noise is the important aspect here rather than the shape of a distinct signal.

Thanks for asking this critical question, it improves the article!

All the best,

Chris

Hi, Chris! Excellent article!!

I’m a beginner in neural networks and I cant’ tell you how much this article has been of great help for me (especially considering the lack of non-academic-level documantation on the topic).

I was just wondering, Tensorflow/Keras has the option to use 1D convolutional layers. Could this project be adapted to use those? and if so, how would one go about doing that? Would it actually benefit from such a change? I’d love to hear your thoughts on the matter!

Hi Andrés,

Thank you for your compliment. Very much appreciated. You can definitely use 1D Convolutional layers. I adapted the article to accomodate for these.

Best,

Chris

Can I link my data from Matlab Simulink with the python ?

Hi Han,

Sure – take a look at this: https://www.mathworks.com/matlabcentral/answers/3082-is-it-possible-to-run-python-code-in-matlab

Best,

Chris

Hi Chris,

thank you for your very interesting tutorial.

I tried to use it to create a denoising of physics signals acquired by detectors, but I found it difficult to adapt it to my case. Can I send you my code for a quick revision?

Tnx

Hi Gianfranco,

Thanks for your compliment. Unfortunately, I do not have time to help people individually 🙁

Best,

Chris

This is interesting. It may help to filter noise of analog waveform from detector (after preamplifier ans shapers).

Hello Chris,

Thanks for the great tutorial.

Can i save this as model after training, reuse the model when i have to perform the denoising of data.If yes, how should i go about saving this trained model

Hi Huma,

Thanks! And yes: https://www.machinecurve.com/index.php/2020/02/14/how-to-save-and-load-a-model-with-keras/

Best,

Chris

HEllo Chris,

Can i use the same approach and feed in accelerometer data which has x,y,z values to train and generate the model and reuse this model to remove the noise from this data?

Hi humayra,

Yes, with some changes to your input shape that should be possible. You do need to know what distribution your noise has by investigating the source of your noise.

Best,

Chris

Hello Chris,

Thanks for your great tutorial.

It seems that someone else also found it difficult to test the model on different signals.

Unlike 2D pictures, it is really hard to design the 1D signal training dataset.

Therefore, do you think the denoising autoencoder cannot be applied to real data?

Hello Chris,

Thanks for your great tutorial.

It seems that someone else also found it difficult to test the model on different signals.

Unlike 2D pictures, it is really hard to design the 1D signal training dataset.

Therefore, do you think the denoising autoencoder cannot be applied to real data?

Hi Rain,

Thank you for your compliment.

I would not necessarily agree 🙂

You can generate a noisy signal by taking a variety of functions, adding noise to them, and then sampling from hundreds of thousands small domain intervals — this will yield a variety of shapes, but all noisy.

You then also take the non-noisy corresponding signals, and train the autoencoder.

Perhaps, this helps it generalize much better.

Best regards,

Chris