Machine Learning Glossary

Machine Learning, Computer Vision, and Computer Graphics Glossary

A

• Attention - An element used in transformers which involves computing the product of query embeddings and key embeddings to compute the interaction between sequence elements.

B

• Backwards propagation - Also known as backprop or backpropagation. Application of the chain rule on neural networks to compute gradients for each parameters. Known as backpropagation because you need to know gradients at the following layers to compute each layers gradient.

C

• Capsule neural network
• Convolutional neural network or CNN - A neural network architecture for image data, or other data on a regular grid.

D

• Dilation - Spacing between elements in a CNN kernel when applied. See Convolutional neural network.
• Domain Adaptation - An area of research focused on making neural network work with alternate domains, or sources of data.

E

• Early stopping - a technique where you stop training once the validation loss begins increasing. This is not as used these days with large models.

F

• Forward propagation - Inference through a neural network by computing each layer's outputs.
• Fully connected network - The standard neural network model where each layer is a sequence of nodes.

G

• Generalization - How well a model works on data it has not been trained on.
• Generative adversarial network (GAN) - A neural network setup for generating examples from a training distribution.
• Gradient Descent - The operation used to update parameters when optimizing neural network. Also known as direction of steepest descent.
• Graph neural network (GNN) - A type of neural network which operates on graph inputs.

I

• Intersection over Union (IoU) - A metric for computing the accuracy of bounding box prediction.

L

• L1 or L2 norm - Two common norms used for computing accuracy or losses.
• Loss function - Target function which you are attempting to minimize.
• Long short-term memory (LSTM) - An RNN neural network architecture which has two sets of hidden states for long and short term.

M

• MSE - Mean squared error. The L2 loss without the square root.
• Multilayer perceptron (MLP) - See Fully connected network.

N

• Normalized Device Coordinates - In images, pixels are in coordinates of \(\displaystyle [-1, 1]\times[-1, 1] \).

O

• Overfitting - when a model begins to learn attributes specific to your training data, thereby worsening performance on non-training data.

R

• Recurrent neural network (RNN) - A type of neural network which operates sequentially on sequence data.
• Reinforcement Learning - an area of machine learning focused on learning to perform actions, E.g. playing a game

S

• Stride - how far the CNN kernel in terms of input pixels moves between output pixels.

T

• Transfer Learning - Techniques to make a neural network perform a different task than what it is trained on.
• Transformer (machine learning model) - A neural network architecture for sequence data which uses attention between elements of the sequence.

U

• Underfitting - when a model performs poorly on both training and validation data, usually due to inadequate model complexity or training duration.