Last Updated on 30 March 2021

Recently, we’ve looked at convolutional layers and certain variations to see how they can be used in machine learning problems. Today, we’ll focus on a variant called *transposed convolution*, which can be used for upsampling images (making them larger) or finding the original representation of a convolutional filter map.

We’ll first cover a normal convolution before we introduce transposed ones. We do so by means of the convolution matrix. Hope you’ll enjoy!

After reading this tutorial, you will understand…

- What normal convolutions do.
- How transposed convolutions can be used for reversing the output of your ConvNet.
- Applications of transposed convolutions.

Let’s take a look! 🚀

**Update 09/Feb/2021: **ensure that article is up to date.

**Update 01/Mar/2020: **adapted images for the “normal convolution” to make them equal to the convolution matrix example.

## Table of contents

- Summary: understanding transposed convolutions
- What does a normal convolution do?
- The goal: reconstructing the original input
- Transposed Convolution
- Representing the normal convolution with a Convolution Matrix
- Computing the normal convolution output with a Convolution Matrix
- From output back to the input: the Transposed Convolution
- Implementations in deep learning frameworks: normal convolution, transposed matrix
- Learnable kernels are what make Transposed convolutions different

- Applications of transposed convolutions
- Summary
- References

## Summary: understanding transposed convolutions

Convolutional Neural Networks are used for computer vision projects and can be used to automatically extract features from inputs like photos and videos. These neural networks employ so-called convolutional layers that convolve (slide) over the input image, try to detect patterns, and adapt weights accordingly during the training process – allowing learning to occur.

Sometimes, however, you want the opposite to happen: invert the output of a convolutional layer and reconstruct the original input. This is for example the case with autoencoders, where you use normal convolutions to learn an encoded state and subsequently decode them into the original inputs. If done successfully, the encoded state can be used as a lower-dimensional representation of your input data, for dimensionality reduction.

Transposed convolutional layers can be used for this purpose. Rather than performing interpolation, they learn a set of weights that can be used to reconstruct original inputs. They can be trained jointly with convolutional layers during the training process. In this article, we’ll cover transposed convolutions in more detail. We’ll show you how they work and how they are applied.

## What does a normal convolution do?

If we wish to understand transposed convolutions, we must be able to compare them with something – and that something, in our case, is a normal convolution.

Let’s look at one:

More specifically, we’re looking at a convolution of a one-channel image: this is likely a grayscale image. Normally, you would convolve over multiple channels, and you would likely use multiple kernels. For the sake of simplicity, our image has one channel and we use N = 1 kernels.

It must now follow why the 2×2 kernel produces a 2×2 output when convolving over a 3×3 image. I’ll briefly recap it next.

When the convolution process starts, the kernel is placed at the upper left corner. It performs element-wise multiplications and hence, produces a scalar output (a number) for the overlapping area. It then moves one step to the right, performs the same thing, but then cannot move any further to the right.

## Let's pause for a second! 👩💻

**free**Machine Learning update today! You will learn

**new things**and

**better understand**concepts you already know.

We send emails at least every Friday. Welcome!

It then simply moves one down, if possible, and does the same trick again. Once it can no longer go to the right, it will attempt to move one down, but cannot do so for the simple reason that we’ve already reached the end of the image. The convolution operation then stops. Note that in the first row, two scalar values were produced, as well as in the second row. These two times two scalar values produce the 2×2 output displayed in the image above.

*Note that we assume a stride of 1 in this example.*

If you wish to understand normal convolutions in more detail, I suggest that you take a look at this post before moving on.

## The goal: reconstructing the original input

Now what if your goal is to do the opposite: given a *summary*, i.e. the result of the convolution, reconstructing the original input?

We call this “upsampling”.

Like this:

You have multiple options.

### Traditional interpolation techniques

First and foremost, it is possible to use traditional interpolation techniques to make your image larger. For example, you could use bicubic or bilinear interpolation to achieve the result you’re interested in. However, they’re not too flexible: they simply compute an estimate of the interpolated pixel values based on their surroundings. In the case of making images larger without losing a sense of detail, we might be interested in a *different approach* – one where the *means of interpolation* is learnt based on the target data. Regular and transposed convolutions then enter the spotlight.

### Regular convolutions – or not?

First, as described more lengthily in Dumoulin & Francesco (2016), you can employ a regular convolution operation. This, however, might not be the most efficient route towards reconstructing the original image:

Finally note that it is always possible to emulate a transposed convolution with a direct convolution. The disadvantage is that it usually involves adding many columns and rows of zeros to the input, resulting in a much less efficient implementation.

Dumoulin & Francesco (2016)

(note that the paper gives many examples – it’s a good recommendation if you wish to understand it in even greater detail!)

Let’s now look at our third possible approach: a *transposed convolution.*

## Transposed Convolution

Rather, we must find another way of doing so. Enter the transposed convolution. We’ll discuss this way of working next in a multi-stage fashion. Firstly, we describe how forward and backwards passes are normally covered by a convolutional layer and how they are inverted in a transposed convolution.

Secondly, we represent the normal convolution with a Convolution Matrix – it’s the first step to demonstrating the power of a transposed convolution. Additionally, we’ll compute the normal convolution output based on this matrix to demonstrate that we’ll achieve the same result.

Subsequently, we’ll introduce the transposed convolution based on the Convolution Matrix we defined earlier, and show that it’s indeed possible to reconstruct the original input.

## Never miss new Machine Learning articles ✅

**free**Machine Learning update today! You will learn

**new things**and

**better understand**concepts you already know.

We send emails at least every Friday. Welcome!

### Representing the normal convolution with a Convolution Matrix

Let’s now see how we can represent the normal convolution by means of a Convolution Matrix. We can use this matrix to demonstrate how the Transposed Convolution works (Dumoulin & Visin, 2016).

Suppose that we’re performing convolutions with a 2×2 kernel on a 3×3 input image, like this:

With our understanding of how regular convolutions work, it’s not surprising to find that we’ll end up with a 2×2 output or feature map:

We can also represent this operation as a Convolution Matrix.

**What is a convolution matrix?**

It’s a matrix which demonstrates all positions of the kernel on the original image, like this:

One who looks closely, will notice that each row represents a position of the kernel on top of the image: in the first row, for example, the kernel is positioned at the top left of the image. The {1, 2, 0} at the first row of the convolution matrix therefore represents the effect of the convolution at the first row of the input image. The {2, 1, 0} represents the effect of the convolution at the second row of the input image. Since at this point in time, the convolution is not applied in either the 3rd column or the 3rd row, either the third column value of the first and second row *and* all the third row values are 0.

Note that when the kernel moves to the right in the second iteration, the current position is represented by the second row of the convolution matrix, and so on. The convolution matrix therefore describes the full convolutional operation of the kernel on the input image.

### Computing the normal convolution output with a Convolution Matrix

The convolution matrix can be used to compute the output of a normal convolution. Doing so is really simple, namely, by flattening the input image into a (9×1) feature vector:

It’s possible to represent the 3 x 3 image as an 1 x 9 image instead, which essentially allows you to contain the same amount of data in the same ordering – by breaking the 1 x 9 image apart after each 3rd block and stacking the 1 x 3 blocks together, you’ll arrive at the 3 x 3 image again.

The fun thing is that we can multiply this (9×1) matrix with the (4×9) convolution matrix and hence achieve a (4×9) x (9×1) = (4×1) output:

When turning it around, breaking it into blocks of two and stacking them vertically, we see that it’s the same as what we saw earlier:

We can thus express the convolutional operation by means of a convolution matrix!

### From output back to the input: the Transposed Convolution

Now suppose that this is your input:

While this is your desired output:

## Join hundreds of other learners! 😎

**free**Machine Learning update today! You will learn

**new things**and

**better understand**concepts you already know.

We send emails at least every Friday. Welcome!

Or, in other words: you’re trying to do the opposite – going backwards from a summarized version of the original input to the original input, rather than creating the summary.

We’ll now find why we call the type of convolution *transposed*, as we can also represent this by means of the convolution matrix – although not the original one, but its transpose:

a.k.a. the one where the columns have become rows and vice-versa.

The input can also be represented as follows:

You may now have noticed that once again, we’re in a situation in which we can perform a matrix multiplication: we now have a (9×4) matrix and a (4×1) matrix, which we can multiply to arrive at a (9×1) matrix or, when broken apart, the (3×3) matrix we were looking for!

### Implementations in deep learning frameworks: normal convolution, transposed matrix

There are effectively two ways of implementing the transposed convolution (Theano, n.d.):

- By applying a regular convolution with many zeroes in padding, to effectively add a
*fractional*stride – however, this is very inefficient as you’ll substantially increase the number of multiplications that are necessary to find your results. - By applying a regular convolution, however also swapping the so-called
*forward*and*backwards pass*. That is, in a normal convolution, when data is fed forward, the output shape gets smaller due to the properties of such convolutions. However, when you swap them, you’ll effectively compute the backwards pass – and make the input larger. This*swap*can be achieved by transposing the convolution matrix, indeed similar as to what we just saw above! Hence, also, the name*transposed*convolutions.

Frameworks such as TensorFlow and Theano implement transposed convolutions in this way or by using a very similar one.

### Learnable kernels are what make Transposed convolutions different

Now, one may wonder:

**Why should I use transposed convolutions rather than traditional interpolation techniques?**

Although this is slightly dependent on why you intend to use such convolutions, there may be very good reasons for doing so.

Note that in both the **regular convolution matrix** and the **transposed one**, the non-zero fields are determined by the *kernel*.

And the kernel is *learnt over time*, as during model optimization they are adapted continuously to better reflect the relationships underlying your dataset.

What this means is that, contrary to regular interpolation techniques, you can *learn* kernels first (e.g. by applying regular convolution operations on your data), and subsequently using them to define your *transposed convolution*, in order to find your original data – or at least, something that hopefully looks like it.

This, in itself, allows one to use them for very interesting applications.

## Applications of transposed convolutions

Firstly, it is of course possible to perform upsampling operations with transposed convolutions. That is, when you have a smaller image, you can make it larger by applying transposed convolutions. Note that you’ll have to learn the weights in order to do so – which might mean that you’re better off with traditional interpolation if you want fast results.

## We help you with Machine Learning! 🧠

**free**Machine Learning update today! You will learn

**new things**and

**better understand**concepts you already know.

We send emails at least every Friday. Welcome!

Secondly, transposed convolutions can be used in Generative Adversarial Networks (Shibuya, 2019). Those networks randomly generate a small matrix and use fractionally-strided convolutions (another name to describe transposed convolutions, but then perhaps in the relatively inefficient implementation of regular convolutions with fractional strides) to upsample them to true images. The weights that have been learnt in the process allow for the upsampling of the random noise to the actual image.

Thirdly, they’re also used in *segmantic segmentation*, where you wish to classify each pixel of an image into a certain category (Shibuya, 2019). They work by generating predictions for intermediate results achieved with convolutions, subsequently upsamling those to find predictions for the originally-shaped input image.

Finally, and perhaps more recently, they are used in what is called a *convolutional autoencoder*. In those, convolutional layers are used to find an *encoding* for some input, i.e. a representation of the original input in much lower dimensionality. A clear example would be a radar image with a landmine and one without a landmine; for the latter, one could train an autoencoder to find a particular encoding. However, autoencoders also contain a *decoding* side: given some encoded input, it attempts to find the original output (by, unsurprisingly, upsampling layers such as transposed convolutions). By comparing the original input image with the output image, it’s possible to identify whether the input belongs to a certain class. In our case, that would be landmine yes/no, and ‘yes’ would only be the case if the decoded encoding really looks like the original image, since then the landmine-trained encoding worked well.

## Summary

In this blog, we’ve seen how transposed convolutions work and what they can be used for. We’ve looked at convolution matrices that can be used for upsampling and demonstrated how both the regular convolution matrix and the transposed one are defined by the kernels that can be learnt. Interesting application areas are upsampling, GANs, semantic segmentation and autoencoders.

Thank you for reading! I hope this blog has made things a bit more clear 🙂 Would you kindly let me know in the comments? 👇 If not – or if you have questions or remarks – please feel free to write a comment as well! I’m more than happy to reply. Should I’ve made a mistake, which can happen, I’ll gladly adapt my blog and note you as a contributor. Thank you!

## References

Dumoulin, Vincent, en Francesco Visin. “A guide to convolution arithmetic for deep learning”. arXiv:1603.07285 [cs, stat], March 2016. arXiv.org, http://arxiv.org/abs/1603.07285.

Theano. (n.d.). Convolution arithmetic tutorial — Theano 1.0.0 documentation. Retrieved from http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html#transposed-convolution-arithmetic

Shibuya, N. (2019, January 1). Up-sampling with Transposed Convolution. Retrieved from https://medium.com/activating-robotic-minds/up-sampling-with-transposed-convolution-9ae4f2df52d0

Good article – but why not just use the figures of your initial example to reconstruct the input, rather than bringing in new figures? It would have made things even clearer, I think

Hi Bill, thanks for your feedback. I adapted the initial example to make it equal to the one that shows how to reconstruct the input.

I agree with Bill. The two examples have the same kernel but with different input/output. I wonder if you use the first example’s output as second example’s input what will happen.

Also I didn’t get the answer “adapted the initial example to make it equal to the one that shows how to reconstruct the input”.

Hi Feng,

I get it now. I thought the first example was meant in the sense that the example at the top was different, which I changed.

But I agree – the two examples (regular Conv and Transpose Conv) are different. If you used the same kernel for Conv and Transpose Conv, you’d get a different result. The operation can be used to maintain the shape of the original input, but does not produce the exact same input with the same kernel. So it’s the connectivity pattern that is established, rather than a 1-to-1 connection.

The weights of a `Conv2DTranapose` layer, for example, are also learnable, so if your goal is to create something that approaches the input as the output (i.e., an autoencoder such as https://www.machinecurve.com/index.php/2019/12/19/creating-a-signal-noise-removal-autoencoder-with-keras/), you’ll use both Conv2D and Conv2DTranspose. Altogether, the weights learned by the Conv and ConvTranspose layers will allow you to approach the original input, but if you inspect the weights of e.g. the first Conv and last ConvTranspose layer (the symmetric layers), you’ll find that the weights are different.

Regards,

Chris