Machine Learning Glossary

Machine Learning, Computer Vision, and Computer Graphics Glossary

A

Attention - An component of transformers which involves computing the product of query embeddings and key embeddings to compute the interaction between elements.

Backwards propagation - Also known as backprop or backpropagation. Application of the chain rule on neural networks to compute gradients for each parameters. Known as backpropagation because you need to know gradients at the following layers to compute each layers gradient.

Capsule neural network
Convolutional neural network or CNN - A neural network architecture for image data, or other data on a regular grid.

Decision Tree - A simple classifier which consists of layers of smaller binary classifiers which each reduces entropy in their classified sets.
Deep Learning - The use of neural networks (i.e. >= 2 layers) in machine learning tasks.
Dilation - Spacing between elements in a CNN kernel when applied. See Convolutional neural network.
Domain Adaptation - An area of research focused on making neural network work with alternate domains, or sources of data.
Dropout - A technique where you zero out the features outputs of a random percent of neurons in each iteration, turning your network into an ensemble of subnetworks during training.

Early stopping - a technique where you stop training once the validation loss begins increasing. This is not as popular these days with large models.
Early exitting - a technique to optimize neural network inference by routing features directly to an output head instead of through the entire model.
Embedding - see latent code. This is a just a feature used to represent something.

Forward propagation - Inference through a neural network by computing each layer's outputs.
Fully connected network - The standard neural network model where each layer is a sequence of nodes.
Features or Feature map - a generic term indicating the intermediate outputs of a neural network

Generalization - How well a model works on data it has not been trained on.
Generative adversarial network (GAN) - A neural network setup for generating examples from a training distribution.
Gradient Descent - The operation used to update parameters when optimizing neural network. Also known as direction of steepest descent.
Graph neural network (GNN) - A type of neural network which operates on graph inputs.

Hinge Loss - A loss used for training classifiers which returns 0 for correct classifications and for bad classifications. \(l=\max(0, 1-y*\hat{y})\)
Hidden Layer - Intermediate layers in a neural network whose outputs are passed to other parts of the neural network.
Hyperparameter - Parameters of a model which are typically hand chosen and not directly optimized during training.

Intersection over Union (IoU) - A metric for computing the accuracy of bounding box prediction.

L1 or L2 norm - Two common norms used for computing accuracy or losses.
Latent code - an input to a neural network which gets optimized during training to represent a particular state or value
Loss function - Target function which you are attempting to minimize.
Long short-term memory (LSTM) - An RNN neural network architecture which has two sets of hidden states for long and short term.

Neurons - Individual elements in a MLP layer (perceptron + activation) which supposedly resemble brain neurons.
Normalized Device Coordinates - In images, pixels are in coordinates of \(\displaystyle [-1, 1]\times[-1, 1] \).

Overfitting - when a model begins to learn attributes specific to your training data, thereby worsening performance on non-training data.

Perceptron - a linear classifier.
Positional encoding - Applying sin/cos at various frequencies (i.e. fourier basis) so the network can distinguish input values at different scales. Used in NeRF as well as in NLP models to indicate the relative position of tokens.

Random Forest - an ensemble learning method which involves building several decision trees, each with different subsets of features.
Recurrent neural network (RNN) - A type of neural network which operates sequentially on sequence data.
Reinforcement Learning - an area of machine learning focused on learning to perform actions, E.g. playing a game
Receptive Field - in a CNN, the size of pixels in the input which affect a value in the output feature map

Stride - how far the CNN kernel in terms of input pixels moves between output pixels.
Support Vector Machine (SVM) - A linear classifier which maximizes the margin/distance to the nearest examples.

Transfer Learning - Techniques to make a neural network perform a different task than what it is trained on.
Transformer (machine learning model) - A neural network architecture which uses attention between elements. Originally designed for NLP, but now used for many other areas.

Underfitting - when a model performs poorly on both training and validation data, usually due to inadequate model complexity or training duration.