Batch normalization: Difference between revisions
No edit summary |
|||
| Line 1: | Line 1: | ||
==Batch Norm in CNNs== | |||
See [https://stackoverflow.com/questions/38553927/batch-normalization-in-convolutional-neural-network Batch norm in CNN]. | |||
In a CNN, the mean and standard deviation are calculated across the batch, width, and height of the features. | |||
<pre> | |||
# t is still the incoming tensor of shape [B, H, W, C] | |||
# but mean and stddev are computed along (0, 1, 2) axes and have just [C] shape | |||
mean = mean(t, axis=(0, 1, 2)) | |||
stddev = stddev(t, axis=(0, 1, 2)) | |||
for i in 0..B-1, x in 0..H-1, y in 0..W-1: | |||
out[i,x,y,:] = norm(t[i,x,y,:], mean, stddev) | |||
</pre> | |||
Revision as of 18:32, 15 June 2020
Batch Norm in CNNs
See Batch norm in CNN.
In a CNN, the mean and standard deviation are calculated across the batch, width, and height of the features.
# t is still the incoming tensor of shape [B, H, W, C] # but mean and stddev are computed along (0, 1, 2) axes and have just [C] shape mean = mean(t, axis=(0, 1, 2)) stddev = stddev(t, axis=(0, 1, 2)) for i in 0..B-1, x in 0..H-1, y in 0..W-1: out[i,x,y,:] = norm(t[i,x,y,:], mean, stddev)