Convolutional neural network: Difference between revisions

(10 intermediate revisions by the same user not shown)

Line 34:

[https://pytorch.org/docs/stable/nn.html#convolution-layers Pytorch Convolution Layers]

[https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d Types of convolutions animations]

Here, we will explain 2d convolutions.

Here, we will explain 2d convolutions, also known as cross-correlation.

Suppose we have the following input image:

<pre>

Line 72:

\end{bmatrix}

</math>

Summing up all the elements gives us <math>66</math> which would go in the first index of the output.

Shifting the kernel over all positions of the image gives us the whole output, another 2D image.

The formula for the output resolution of a convolution is:

Line 86:

Line 87:

===Kernel===

Typically these days people use small kernels e.g. \(3 \times 3\) or \(4 \times 4\).

Typically these days people use small kernels e.g. \(3 \times 3\) or \(4 \times 4\) with many conv layers.

However, historically people used larger kernels (e.g. <math>7 \times 7</math>). This leads to more parameters which need to be trained and thus networks cannot be as deep.

Note that in practice, people use multi-channel inputs so the actual kernel will be 3D.

Line 94:

Line 96:

===Stride===

How much the kernel moves ~~along~~. Typically 1 or 2.

How much the kernel moves. Typically 1 or 2.

Moving by 2 will yield half the resolution of the input.

===Padding===

Line 102:

Line 105:

;Common Types of padding

* Zero/Constant

* Zero or Constant padding

* Mirror/Reflection

* Mirror/Reflection padding

* Replication

* Replication padding

With convolution layers in ~~tensorflow and other~~ libraries you often see these two types of padding:

With convolution layers in libraries you often see these two types of padding which can be added to the conv layer directly:

* <code>VALID</code> - Do not do any padding

* <code>SAME</code> - Apply zero padding such that the output will have resolution \(\lfloor x/stride \rfloor\).

Line 139:

Line 142:

Pooling is one method of reducing and increasing the resolution of your feature maps.

You can also use bilinear upsampling or downsampling.

Typically the stride of pooling is equal to the filter size so a <math>2 \times 2</math> pooling will have a stride of <math>2</math> and result in an image with half the width and height.

===Avg Pooling===

Take the average over a region

Take the average over a region.

This is equivalent to bilinear downsampling.

===Max Pooling===

Line 153:

Line 158:

==Spherical Images==

There are many ways to adapt convolutional layers to spherical images.

~~Below are just a few that I've seen.~~

* [http://papers.nips.cc/paper/6656-learning-spherical-convolution-for-fast-features-from-360-imagery Learning Spherical Convolution for Fast Features from 360 Imagery (NIPS 2017)] proposes using different kernels with different weights and sizes for different altitudes \(\phi\).

@@ Line 34: / Line 34: @@
 [https://pytorch.org/docs/stable/nn.html#convolution-layers Pytorch Convolution Layers]<br>
 [https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d Types of convolutions animations]<br>
-Here, we will explain 2d convolutions.<br>
+Here, we will explain 2d convolutions, also known as cross-correlation.<br>
 Suppose we have the following input image:<br>
 <pre>
@@ Line 72: / Line 72: @@
 \end{bmatrix}
 </math><br>
 Summing up all the elements gives us <math>66</math> which would go in the first index of the output.
+Shifting the kernel over all positions of the image gives us the whole output, another 2D image.
 The formula for the output resolution of a convolution is:
@@ Line 86: / Line 87: @@
 ===Kernel===
-Typically these days people use small kernels e.g. \(3 \times 3\) or \(4 \times 4\).
+Typically these days people use small kernels e.g. \(3 \times 3\) or \(4 \times 4\) with many conv layers.
+However, historically people used larger kernels (e.g. <math>7 \times 7</math>). This leads to more parameters which need to be trained and thus networks cannot be as deep.
 Note that in practice, people use multi-channel inputs so the actual kernel will be 3D.
@@ Line 94: / Line 96: @@
 ===Stride===
-How much the kernel moves along. Typically 1 or 2.
+How much the kernel moves. Typically 1 or 2.
+Moving by 2 will yield half the resolution of the input.
 ===Padding===
@@ Line 102: / Line 105: @@
 ;Common Types of padding
-* Zero/Constant
+* Zero or Constant padding
-* Mirror/Reflection
+* Mirror/Reflection padding
-* Replication
+* Replication padding
-With convolution layers in tensorflow and other libraries you often see these two types of padding:
+With convolution layers in libraries you often see these two types of padding which can be added to the conv layer directly:
 * <code>VALID</code> - Do not do any padding
 * <code>SAME</code> - Apply zero padding such that the output will have resolution \(\lfloor x/stride \rfloor\).
@@ Line 139: / Line 142: @@
 Pooling is one method of reducing and increasing the resolution of your feature maps.
 You can also use bilinear upsampling or downsampling.
+Typically the stride of pooling is equal to the filter size so a <math>2 \times 2</math> pooling will have a stride of <math>2</math> and result in an image with half the width and height.
 ===Avg Pooling===
-Take the average over a region
+Take the average over a region.
+This is equivalent to bilinear downsampling.
 ===Max Pooling===
@@ Line 153: / Line 158: @@
 ==Spherical Images==
 There are many ways to adapt convolutional layers to spherical images.
-Below are just a few that I've seen.
 * [http://papers.nips.cc/paper/6656-learning-spherical-convolution-for-fast-features-from-360-imagery Learning Spherical Convolution for Fast Features from 360 Imagery (NIPS 2017)] proposes using different kernels with different weights and sizes for different altitudes \(\phi\).