Batch normalization: Difference between revisions

From David's Wiki
No edit summary
Line 1: Line 1:
==Batch Norm in CNNs==
See [https://stackoverflow.com/questions/38553927/batch-normalization-in-convolutional-neural-network Batch norm in CNN].
In a CNN, the mean and standard deviation are calculated across the batch, width, and height of the features.
<pre>
# t is still the incoming tensor of shape [B, H, W, C]
# but mean and stddev are computed along (0, 1, 2) axes and have just [C] shape
mean = mean(t, axis=(0, 1, 2))
stddev = stddev(t, axis=(0, 1, 2))
for i in 0..B-1, x in 0..H-1, y in 0..W-1:
  out[i,x,y,:] = norm(t[i,x,y,:], mean, stddev)
</pre>





Revision as of 18:32, 15 June 2020

Batch Norm in CNNs

See Batch norm in CNN.

In a CNN, the mean and standard deviation are calculated across the batch, width, and height of the features.

# t is still the incoming tensor of shape [B, H, W, C]
# but mean and stddev are computed along (0, 1, 2) axes and have just [C] shape
mean = mean(t, axis=(0, 1, 2))
stddev = stddev(t, axis=(0, 1, 2))
for i in 0..B-1, x in 0..H-1, y in 0..W-1:
  out[i,x,y,:] = norm(t[i,x,y,:], mean, stddev)


Resources