Creating a Signal Noise Removal Autoencoder with Keras

Pure signals only exist in theory. That is, when you’re doing signal processing related activities, it’s very likely that you’ll experience noise. Whether noise is caused by the measurement (or reception) device or by the medium in which you perform measurements, you want it gone.

Various mathematical tricks exist to filter out noise from a signal. When noise is relatively constant across a range of signals, for example, you can take the mean of all the signals and deduct it from each individual signal – which likely removes the factors that contribute from noise.

However, these tricks work by knowing a few things about the noise up front. In many cases, the exact shape of your noise is unknown or cannot be estimated because it is relatively hidden. In those cases, the solution may lie in learning the noise from example data.

Autoencoders can be used for this purpose. By feeding them noisy data as inputs and clean data as outputs, it’s possible to make them recognize the ideosyncratic noise for the training data. This way, autoencoders can serve as denoisers.

But what are autoencoders exactly? And why does the way they work make them suitable for noise removal? And how to implement one for signal denoising / noise reduction?

We’ll answer these questions in today’s blog. First, we’ll provide a recap on autoencoders – to (re)gain a theoretical understanding of what they are and how they work. This includes a discussion on why they can be applied to noise removal. Subsequently, we implement an autoencoder to demonstrate this, by means of a three-step process:

• We generate a large dataset of \(x^2\) samples.
• We generate a large dataset of \(x^2\) samples to which Gaussian (i.e., random) noise has been added.
• We create an autoencoder which learns to transform noisy \(x^2\) inputs into the original sine, i.e. which removes the noise – also for new data!

Ready?

Okay, let’s go! 😊

Recap: what are autoencoders?

If we’re going to build an autoencoder, we must know what they are.

In our blog post “Conv2DTranspose: using 2D transposed convolutions with Keras”, we already covered the high-level principles behind autoencoders, but it’s wise to repeat them here.

We can visualize the flow of an autoencoder as follows:

Autoencoders are composed of two parts: an encoder, which encodes some input into encoded state, and a decoder which can decode the encoded state into another format. This can be a reconstruction of the original input, as we can see in the plot below, but it can also be something different.

For example, autoencoders are learnt for noise removal, but also for dimensionality reduction (Keras Blog , n.d.; we then use them to convert the input data into low-dimensional format, which might benefit training lower-dimensionality model types such as SVMs).

Note that the red parts in the block above – that is, the encoder and the decoder, are learnt based on data (Keras Blog, n.d.). This means that, contrary to more abstract mathematical functions (e.g. filters), they are highly specialized in one domain (e.g. signal noise removal at \(x^2\) plots as we will do next) while they perform very poorly in another (e.g. when using the same autoencoder for image noise removal).

Why autoencoders are applicable to noise removal

Autoencoders learn an encoded state with an encoder, and learn to decode this state into something else with a decoder.

Now think about this in the context of signal noise: suppose that you feed the neural network noisy data as features, while you have the pure data available as targets. Following the drawing above, the neural network will learn an encoded state based on the noisy image, and will attempt to decode it to best match the pure data. What’s the thing that stands in between the pure data and the noisy data? Indeed, the noise. In effect, the autoencoder will thus learn to recognize noise and remove it from the input image.

Let’s now see if we can create such an autoencoder with Keras.

Today’s example: a Keras based autoencoder for noise removal

In the next part, we’ll show you how to use the Keras deep learning framework for creating a denoising or signal removal autoencoder. Here, we’ll first take a look at two things – the data we’re using as well as a high-level description of the model.

The data

First, the data. As pure signals (and hence autoencoder targets), we’re using pure \(x^2\) samples from a small domain. When plotted, a sample looks like this:

For today’s model, we use 100.000 samples. To each of them, we add Gaussian noise – or random noise. While the global shape remains present, it’s clear that the plots become noisy:

The model

Now, the model. It looks as follows:

…and has these layers:

• The input layer, which takes the input data;
• Two Conv2D layers, which serve as encoder;
• Two Conv2D transpose layers, which serve as decoder;
• One Conv2D layer with one output, a Sigmoid activation function and padding, serving as the output layer.

You might now wonder – why Conv2D?

This is a valid question, given the fact that our \(x^2\) data can technically be handled with a one-dimensional Conv layer, i.e. Conv1D.

The answer is simple: there is no such thing as Conv1DTranspose in Keras as of December 2019 😀 I do however favor transposed convolutions very much when creating autoencoders, for their simplicity of use – especially compared to upsampling with regular convolutions, which would have been possible with UpSampling1D.

Hence, I’m using Conv2D.

Yes, indeed: this has impact on the data. I’m actually reshaping the data into two-dimensional format before feeding it to the model, then to resample the prediction into onedimensional format for visualization purposes. As we will see, this works flawlessly.

To provide more details, this is the model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 13, 8, 128)        1280
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 11, 6, 32)         36896
_________________________________________________________________
conv2d_transpose_1 (Conv2DTr (None, 13, 8, 32)         9248
_________________________________________________________________
conv2d_transpose_2 (Conv2DTr (None, 15, 10, 128)       36992
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 15, 10, 1)         1153
=================================================================
Total params: 85,569
Trainable params: 85,569
Non-trainable params: 0
_________________________________________________________________
Train on 56000 samples, validate on 14000 samples

Let’s now start with the first part – generating the pure waveforms! Open up your Explorer, navigate to some folder (e.g. keras-autoencoders) and create a file called signal_generator.py. Next, open this file in your code editor – and let the coding process begin!

Generating pure waveforms

Generating pure waveforms consists of the following steps, in order to generate visualizations like the one shown on the right:

• Adding the necessary imports to the start of your Python script;
• Defining configuration settings for the signal generator;
• Generating the data, a.k.a. the pure waveforms;
• Saving the waveforms and visualizing a subset of them.

Adding imports

First, the imports – it’s a simple list:

import matplotlib.pyplot as plt
import numpy as np

We use Numpy for data generation & processing and Matplotlib for visualizing some of the samples at the end.

Configuring the generator

Generator configuration consists of three steps: sample-wide configuration, intra-sample configuration and other settings. First, sample-wide configuration, which is just the number of samples to generate:

# Sample configuration
num_samples = 100000

Followed by intra-sample configuration:

# Intrasample configuration
num_elements = 1
interval_per_element = 0.01
total_num_elements = int(num_elements / interval_per_element)
starting_point = int(0 - 0.5*total_num_elements)

num_elements represents the width of your domain. interval_per_element represents the step size that the iterator will take when generating the sample. In this case, the domain \((0, 1]\) will thus contain 100 samples (as \(1/interval per element = 1/0.01 = 100\)). That’s what’s represented in total_num_elements.

The starting point determines where to start the generation process.

Finally, you can set the number of samples that you want visualized in the other configuration settings:

# Other configuration
num_samples_visualize = 1

Generating data

Next step, creating some data! 😁

We’ll first specify the lists that contain our data and the sub-sample data (one sample in samples contains multiple xs and ys; when \(totalnumelements = 100\), that will be 100 of them each):

# Containers for samples and subsamples
samples = []
xs = []
ys = []

Next, the actual data generation part:

# Generate samples
for j in range(0, num_samples):
  # Report progress
  if j % 100 == 0:
    print(j)
  # Generate wave
  for i in range(starting_point, total_num_elements):
    x_val = i * interval_per_element
    y_val = x_val * x_val
    xs.append(x_val)
    ys.append(y_val)
  # Append wave to samples
  samples.append((xs, ys))
  # Clear subsample containers for next sample
  xs = []
  ys = []

We’ll first iterate over every sample, determined by the range between 0 and the num_samples variable. This includes a progress report every 100 samples.

Next, we construct the wave step by step, adding the function outputs to the xs and ys variables.

Subsequently, we append the entire wave to the samples list, and clear the subsample containers for generating the next sample.

Saving & visualizing

The next step is to save the data. We do so by using Numpy’s save call, and save samples to a file called ./signal_waves_medium.py.

# Input shape
print(np.shape(np.array(samples[0][0])))
  
# Save data to file for re-use
np.save('./signal_waves_medium.npy', samples)

# Visualize a few random samples
for i in range(0, num_samples_visualize):
  random_index = np.random.randint(0, len(samples)-1)
  x_axis, y_axis = samples[random_index]
  plt.plot(x_axis, y_axis)
  plt.title(f'Visualization of sample {random_index} ---- y: f(x) = x^2')
  plt.show()

Next, with some basic Matplotlib code, we visualize num_samples_visualize random samples from the samples array. And that’s it already!

Run your code with python signal_generator.py (ensure that you have Numpy and Matplotlib installed) and the generation process should begin, culminating in a .npy file and one (or more) visualizations popping up once the process finishes.

Full generator code

If you wish to obtain the entire signal generator at once, here you go:

import matplotlib.pyplot as plt
import numpy as np

# Sample configuration
num_samples = 100000

# Intrasample configuration
num_elements = 1
interval_per_element = 0.01
total_num_elements = int(num_elements / interval_per_element)
starting_point = int(0 - 0.5*total_num_elements)

# Other configuration
num_samples_visualize = 1

# Containers for samples and subsamples
samples = []
xs = []
ys = []

# Generate samples
for j in range(0, num_samples):
  # Report progress
  if j % 100 == 0:
    print(j)
  # Generate wave
  for i in range(starting_point, total_num_elements):
    x_val = i * interval_per_element
    y_val = x_val * x_val
    xs.append(x_val)
    ys.append(y_val)
  # Append wave to samples
  samples.append((xs, ys))
  # Clear subsample containers for next sample
  xs = []
  ys = []

# Input shape
print(np.shape(np.array(samples[0][0])))
  
# Save data to file for re-use
np.save('./signal_waves_medium.npy', samples)

# Visualize a few random samples
for i in range(0, num_samples_visualize):
  random_index = np.random.randint(0, len(samples)-1)
  x_axis, y_axis = samples[random_index]
  plt.plot(x_axis, y_axis)
  plt.title(f'Visualization of sample {random_index} ---- y: f(x) = x^2')
  plt.show()

Adding noise to pure waveforms

The second part: adding noise to the 100k pure waveforms we generated in the previous step.

It’s composed of these individual steps:

• Once again, adding imports;
• Setting the configuration variables for the noising process;
• Loading the data;
• Adding the noise;
• Saving the noisy samples and visualizing a few of them.

Create an additional file, e.g. signal_apply_noise.py, and let’s add the following things.

Adding imports

Our imports are the same as we used in the signal generator:

import matplotlib.pyplot as plt
import numpy as np

Configuring the noising process

Our noising configuration is also a lot simpler:

# Sample configuration
num_samples_visualize = 1
noise_factor = 0.05

num_samples_visualize is the number of samples we wish to visualize once the noising process finishes, and noise_factor is the amount of noise we’ll be adding to our samples (\(0 = no noise; 1 = full noise\)).

Loading data

Next, we load the data and assign the samples to the correct variables, being x_val and y_val.

# Load data
data = np.load('./signal_waves_medium.npy')
x_val, y_val = data[:,0], data[:,1]

Adding noise

Next, we add the noise to our samples.

# Add noise to data
noisy_samples = []
for i in range(0, len(x_val)):
  if i % 100 == 0:
    print(i)
  pure = np.array(y_val[i])
  noise = np.random.normal(0, 1, pure.shape)
  signal = pure + noise_factor * noise
  noisy_samples.append([x_val[i], signal])

First, we define a new list that will contain our noisy samples. Subsequently, we iterate over each sample (reporting progress every 100 samples). We then do a couple of things:

• We assign the pure sample (i.e., the \(x^2\) plot wihtout noise) to the pure variable.
• Subsequently, we generate Gaussian noise using np.random.normal, with the same shape as pure‘s.
• Next, we add the noise to the pure sample, using the noise_factor.
• Finally, we append the sample’s domain and the noisy sample to the noisy_samples array.

Saving & visualizing

Next, we – and this is no different than with the generator before – save the data into a .npy file (this time, with a different name 😃) and visualize a few random samples based on the number you configured earlier.

# Save data to file for re-use
np.save('./signal_waves_noisy_medium.npy', noisy_samples)

# Visualize a few random samples
for i in range(0, num_samples_visualize):
  random_index = np.random.randint(0, len(noisy_samples)-1)
  x_axis, y_axis = noisy_samples[random_index]
  plt.plot(x_axis, y_axis)
  plt.title(f'Visualization of noisy sample {random_index} ---- y: f(x) = x^2')
  plt.show()

If you would now run signal_apply_noise.py, you’d get 100k noisy samples, with which we can train the autoencoder we’ll build next.

Full noising code

If you’re interested in the full code of the noising script, here you go:

import matplotlib.pyplot as plt
import numpy as np

# Sample configuration
num_samples_visualize = 1
noise_factor = 0.05

# Load data
data = np.load('./signal_waves_medium.npy')
x_val, y_val = data[:,0], data[:,1]

# Add noise to data
noisy_samples = []
for i in range(0, len(x_val)):
  if i % 100 == 0:
    print(i)
  pure = np.array(y_val[i])
  noise = np.random.normal(0, 1, pure.shape)
  signal = pure + noise_factor * noise
  noisy_samples.append([x_val[i], signal])
  
# Save data to file for re-use
np.save('./signal_waves_noisy_medium.npy', noisy_samples)

# Visualize a few random samples
for i in range(0, num_samples_visualize):
  random_index = np.random.randint(0, len(noisy_samples)-1)
  x_axis, y_axis = noisy_samples[random_index]
  plt.plot(x_axis, y_axis)
  plt.title(f'Visualization of noisy sample {random_index} ---- y: f(x) = x^2')
  plt.show()

Creating the autoencoder

It’s now time for the interesting stuff: creating the autoencoder 🤗

Creating it contains these steps:

• Once again, adding some imports 😋
• Setting configuration details for the model;
• Data loading and preparation;
• Defining the model architecture;
• Compiling the model and starting training;
• Visualizing denoised waveforms from the test set, to find out visually whether it works.

To run it successfully, you’ll need Keras (and by consequence Python and one of the Keras backends, preferably Tensorflow), Matplotlib and Numpy.

Let’s create a third (and final 😋) file: python signal_autoencoder.py.

Adding imports

First, let’s specify the imports:

import keras
from keras.models import Sequential
from keras.layers import Conv2D, Conv2DTranspose
from keras.constraints import max_norm
import matplotlib.pyplot as plt
import numpy as np
import math

From Keras, we import the Sequential API (which we use to stack the layers on top of each other), the Conv2D and Conv2DTranspose layers (see the architecture and the rationale here to find out why), and the MaxNorm constraint, in order to keep the weight updates in check. We also import Matplotlib, Numpy and the Python math library.

Model configuration

Next, we set some configuration options for the model:

# Model configuration
width, height = 15, 10
input_shape = (width, height, 1)
batch_size = 150
no_epochs = 5
train_test_split = 0.3
validation_split = 0.2
verbosity = 1
max_norm_value = 2.0

Please recall that we use two-dimensional Conv layers instead of one-dimensional ones (which are native to our data) because of the availability of Conv2DTranspose. Given the shape of our generated data of \((150, )\), this impacts our configuration in multiple ways:

• We have to specify a weight and height as if the sample is an image, but have to take into account that weight * height must be 150. Hence, we use 15 and 10 for them, respectively.
• The input_shape, in line with Conv2D layers and their three dimensions (image width, image height and the number of channels), is thus \((width, height, 1) = (15, 10, 1)\).
• The batch size is 150. This number seemed to work well, offering a nice balance between loss value and prediction time.
• The number of epochs is fairly low, but pragmatic: the autoencoder did not improve substantially anymore after this number.
• We use 30% of the total data, i.e. 30k samples, as testing data.
• 20% of the training data (70k) will be used for validation purposes. Hence, 14k will be used to validate the model per epoch (and even per minibatch), while 56k will be used for actual training.
• All model outputs are displayed on screen, with verbosity mode set to True.
• The max_norm_value is 2.0. This value worked well in a different scenario, and slightly improved the training results.

Data loading & preparation

The next thing to do is to load the data. We load both the noisy and the pure samples into their respective variables:

# Load data
data_noisy = np.load('./signal_waves_noisy_medium.npy')
x_val_noisy, y_val_noisy = data_noisy[:,0], data_noisy[:,1]
data_pure = np.load('./signal_waves_medium.npy')
x_val_pure, y_val_pure = data_pure[:,0], data_pure[:,1]

Next, we’ll reshape the data. We do so for each sample. This includes the following steps:

• First, given the way how binary crossentropy loss works, we normalize our samples to fall in the range \([0, 1]\). Without this normalization step, odd loss values (extremely negative ones, impossible with BCE loss) start popping up (Quetzalcohuatl, n.d.).
• Next, we resample each sample (currently with shape \((150, )\)) into two-dimensional shape (\((15, 10)\)).
• We subsequently add the resampled noisy and pure samples to the specific *_r arrays.

# Reshape data
y_val_noisy_r = []
y_val_pure_r = []
for i in range(0, len(y_val_noisy)):
  noisy_sample = y_val_noisy[i]
  pure_sample = y_val_pure[i]
  noisy_sample = (noisy_sample - np.min(noisy_sample)) / (np.max(noisy_sample) - np.min(noisy_sample))
  pure_sample = (pure_sample - np.min(pure_sample)) / (np.max(pure_sample) - np.min(pure_sample))
  noisy_sample = noisy_sample.reshape(width, height)
  pure_sample  = pure_sample.reshape(width, height)
  y_val_noisy_r.append(noisy_sample)
  y_val_pure_r.append(pure_sample)
y_val_noisy_r   = np.array(y_val_noisy_r)
y_val_pure_r    = np.array(y_val_pure_r)
noisy_input     = y_val_noisy_r.reshape((y_val_noisy_r.shape[0], y_val_noisy_r.shape[1], y_val_noisy_r.shape[2], 1))
pure_input      = y_val_pure_r.reshape((y_val_pure_r.shape[0], y_val_pure_r.shape[1], y_val_pure_r.shape[2], 1))

Once each sample is resampled, we convert the entire array for both the resampled noisy and resampled pure samples into a structure that Keras can handle. That is, we increase the shape with another dimension to represent the number of channels, which in our case is just 1.

Finally, we perform the split into training and testing data (30k test, 56+14 = 70k train):

# Train/test split
percentage_training = math.floor((1 - train_test_split) * len(noisy_input))
noisy_input, noisy_input_test = noisy_input[:percentage_training], noisy_input[percentage_training:]
pure_input, pure_input_test = pure_input[:percentage_training], pure_input[percentage_training:]

Creating the model architecture

This is the architecture of our autoencoder:

# Create the model
model = Sequential()
model.add(Conv2D(128, kernel_size=(3, 3), kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform', input_shape=input_shape))
model.add(Conv2D(32, kernel_size=(3, 3), kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv2DTranspose(32, kernel_size=(3,3), kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv2DTranspose(128, kernel_size=(3,3), kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv2D(1, kernel_size=(3, 3), kernel_constraint=max_norm(max_norm_value), activation='sigmoid', padding='same'))

model.summary()

• We’ll use the Sequential API, for stacking the layers on top of each other.
• The two Conv2D layers serve as the encoder, and learn 128 and 32 filters, respectively. They activate with the ReLU activation function, and by consequence require He initialization. Max-norm regularization is applied to each of them.
• The two Conv2DTranspose layers, which learn 32 and 128 filters, serve as the decoder. They also use ReLU activation and He initialization, as well as Max-norm regularization.
• The final Conv layer serves as the output layer, and does (by virtue of padding='same') not alter the shape, except for the number of channels (back into 1).
• Kernel sizes are 3 x 3 pixels.

Generating a model summary, i.e. calling model.summary(), results in this summary, which also shows the number of parameters that is trained:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 13, 8, 128)        1280
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 11, 6, 32)         36896
_________________________________________________________________
conv2d_transpose_1 (Conv2DTr (None, 13, 8, 32)         9248
_________________________________________________________________
conv2d_transpose_2 (Conv2DTr (None, 15, 10, 128)       36992
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 15, 10, 1)         1153
=================================================================
Total params: 85,569
Trainable params: 85,569
Non-trainable params: 0
_________________________________________________________________
Train on 56000 samples, validate on 14000 samples

Model compilation & starting the training process

The next thing to do is to compile the model (i.e., specify the optimizer and loss function) and to start the training process. We use Adam and Binary crossentropy for the fact that they are relatively default choices for today’s deep learning models.

Fitting the data shows that we’re going from noisy_input (features) to pure_input (targets). The number of epochs, the batch size and the validation split are as configured earlier.

# Compile and fit data
model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(noisy_input, pure_input,
                epochs=no_epochs,
                batch_size=batch_size,
                validation_split=validation_split)

Visualizing denoised waveforms from test set

Once the training process finishes, it’s time to find out whether our model actually works. We do so by generating a few reconstructions: we add a noisy sample from the test set (which is data the model has never seen before!) and visualize whether it outputs the noise-free shape. This is the code

# Generate reconstructions
num_reconstructions = 4
samples = noisy_input_test[:num_reconstructions]
reconstructions = model.predict(samples)

# Plot reconstructions
for i in np.arange(0, num_reconstructions):
  # Prediction index
  prediction_index = i + percentage_training
  # Get the sample and the reconstruction
  original = y_val_noisy[prediction_index]
  pure = y_val_pure[prediction_index]
  reconstruction = np.array(reconstructions[i]).reshape((width * height,))
  # Matplotlib preparations
  fig, axes = plt.subplots(1, 3)
  # Plot sample and reconstruciton
  axes[0].plot(original)
  axes[0].set_title('Noisy waveform')
  axes[1].plot(pure)
  axes[1].set_title('Pure waveform')
  axes[2].plot(reconstruction)
  axes[2].set_title('Conv Autoencoder Denoised waveform')
  plt.show()

Open up your terminal again, and run python signal_autoencoder.py. Now, the training process should begin.

Full model code

If you’re interested in the full code, here you go:

import keras
from keras.models import Sequential
from keras.layers import Conv2D, Conv2DTranspose
from keras.constraints import max_norm
import matplotlib.pyplot as plt
import numpy as np
import math

# Model configuration
width, height = 15, 10
input_shape = (width, height, 1)
batch_size = 150
no_epochs = 5
train_test_split = 0.3
validation_split = 0.2
verbosity = 1
max_norm_value = 2.0

# Load data
data_noisy = np.load('./signal_waves_noisy_medium.npy')
x_val_noisy, y_val_noisy = data_noisy[:,0], data_noisy[:,1]
data_pure = np.load('./signal_waves_medium.npy')
x_val_pure, y_val_pure = data_pure[:,0], data_pure[:,1]

# Reshape data
y_val_noisy_r = []
y_val_pure_r = []
for i in range(0, len(y_val_noisy)):
  noisy_sample = y_val_noisy[i]
  pure_sample = y_val_pure[i]
  noisy_sample = (noisy_sample - np.min(noisy_sample)) / (np.max(noisy_sample) - np.min(noisy_sample))
  pure_sample = (pure_sample - np.min(pure_sample)) / (np.max(pure_sample) - np.min(pure_sample))
  noisy_sample = noisy_sample.reshape(width, height)
  pure_sample  = pure_sample.reshape(width, height)
  y_val_noisy_r.append(noisy_sample)
  y_val_pure_r.append(pure_sample)
y_val_noisy_r   = np.array(y_val_noisy_r)
y_val_pure_r    = np.array(y_val_pure_r)
noisy_input     = y_val_noisy_r.reshape((y_val_noisy_r.shape[0], y_val_noisy_r.shape[1], y_val_noisy_r.shape[2], 1))
pure_input      = y_val_pure_r.reshape((y_val_pure_r.shape[0], y_val_pure_r.shape[1], y_val_pure_r.shape[2], 1))

# Train/test split
percentage_training = math.floor((1 - train_test_split) * len(noisy_input))
noisy_input, noisy_input_test = noisy_input[:percentage_training], noisy_input[percentage_training:]
pure_input, pure_input_test = pure_input[:percentage_training], pure_input[percentage_training:]

# Create the model
model = Sequential()
model.add(Conv2D(128, kernel_size=(3, 3), kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform', input_shape=input_shape))
model.add(Conv2D(32, kernel_size=(3, 3), kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv2DTranspose(32, kernel_size=(3,3), kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv2DTranspose(128, kernel_size=(3,3), kernel_constraint=max_norm(max_norm_value), activation='relu', kernel_initializer='he_uniform'))
model.add(Conv2D(1, kernel_size=(3, 3), kernel_constraint=max_norm(max_norm_value), activation='sigmoid', padding='same'))

model.summary()

# Compile and fit data
model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(noisy_input, pure_input,
                epochs=no_epochs,
                batch_size=batch_size,
                validation_split=validation_split)

# Generate reconstructions
num_reconstructions = 4
samples = noisy_input_test[:num_reconstructions]
reconstructions = model.predict(samples)

# Plot reconstructions
for i in np.arange(0, num_reconstructions):
  # Prediction index
  prediction_index = i + percentage_training
  # Get the sample and the reconstruction
  original = y_val_noisy[prediction_index]
  pure = y_val_pure[prediction_index]
  reconstruction = np.array(reconstructions[i]).reshape((width * height,))
  # Matplotlib preparations
  fig, axes = plt.subplots(1, 3)
  # Plot sample and reconstruciton
  axes[0].plot(original)
  axes[0].set_title('Noisy waveform')
  axes[1].plot(pure)
  axes[1].set_title('Pure waveform')
  axes[2].plot(reconstruction)
  axes[2].set_title('Conv Autoencoder Denoised waveform')
  plt.show()

Results

Next, the results 😎

After the fifth epoch, validation loss \(\approx 0.3556\). This is high, but acceptable. What’s more important is to find out how well the model works when visualizing the test set predictions.

Here they are:

Clearly, the autoencoder has learnt to remove much of the noise. As you can see, the denoised samples are not entirely noise-free, but it’s a lot better. Some nice results! 😎

Summary

In this blog post, we created a denoising / noise removal autoencoder with Keras, specifically focused on signal processing. By generating 100.000 pure and noisy samples, we found that it’s possible to create a trained noise removal algorithm that is capable of removing specific noise from input data. I hope you’ve learnt something today, and if you have any questions or remarks – please feel free to leave a comment in the comments box below! 😊 I’ll try to answer your comment as soon as I can.

Thanks for reading MachineCurve today and happy engineering! 😎

Please note that the code for these models is also available in my keras-autoencoders Github repository.

References

Keras Blog. (n.d.). Building Autoencoders in Keras. Retrieved from https://blog.keras.io/building-autoencoders-in-keras.html

Quetzalcohuatl. (n.d.). The loss becomes negative · Issue #1917 · keras-team/keras. Retrieved from https://github.com/keras-team/keras/issues/1917#issuecomment-451534575

Share this post:

Linkedin

Facebook

Twitter

Pinterest

email

Print