Visual Learning and Recognition

Notes for CMSC828I Visual Learning and Recognition (Fall 2020) taught by Abhinav Shrivastava

This class covers:

• How a sub-topic evolved
• State of the art

Introduction to Data

Lecture 3 September 8, 2020

The extremes of data. If we have very few images, we are working on an extrapolation problem.
As we approach an infinite number of training samples, learning becomes an interpolation problem.
Traditional datasets are in the order of \(\displaystyle 10^2-10^4\) training samples.
Current datasets are in the order of \(\displaystyle 10^5-10^7\) training samples.

In tiny images , Torrabla et al. use 80 million tiny images.

What is the capacity of visual long term memory?

In Standing (1973), people could recall whether they've seen 10,000 images with 83% recognition.

What we don't know is what people are remembering for each item?

In Brady et al., they tested recall for novel (new object), exemplar (same type of object), and state (same object & state). They got 92% for novel, 88% for exemplar, and 87% for state so humans remember the exact state of objects they've seen.

Rule of thumb

(Simple algorithms + big data) is better than (complicated algorithms + small data)

Lecture 4 (September 10, 2020)

This lecture is on the bias of data. It follows Torralba et al.

Will big data solve all our problems?

E.g. Can (big company) just dump millions of dollars to collect data and solve any problem?
No. E.g. COVID.
There will always be new tasks or problems.

We will never have enough data

Long tails - Zipf's law

Data is biased

Types of visual bias:

• Observer Bias (human vs bird)
• Capture Bias (photographer vs robot)
• Selection Bias (Flickr vs Google Street View)
• Category/Label Bias
• Negative Set Bias

In general, all datasets will have all of these biases mixed in.

• Social Bias

Graduation photos always have a certain structure.

Measuring Dataset Bias

Evaluate cross-dataset performance
Train on one dataset, test on another

To evaluate negative set bias, pool negatives from other datasets (e.g. not car or not person).
They found that models trained on a dataset do ~8% worse at detecting negatives from other datasets.

Overcoming Datset Bias

Mixing datasets

Selection bias

In general, automatically gathered images do better.
You can also collect data from multiple sources (multiple search engines across multiple countries) or collect unannotated images and label them via crowd-sourcing.

Capture bias

To overcome the bias of professional photographs:
Apply data augmentations: flipping images, jittering (small affine transformations), random crops.

Negative set bias

Add negatives from other datasets.
Mine hard negatives from other datasets using standard algorithms.

Data-driven Methods in Vision

Beginning of Lecture 5 (September 15)

Dale et al. perform semantic color correction using a large dataset.
Heys and Efros perform scene completion.
Heys and Efros perform image localization.
Kaneva et al. perform scene matching with camera view transformations.

Will be on the exam

• Back-prop and SGD,
• Softmax, sigmoid, cross entropy