TensorFlow: Difference between revisions

From David's Wiki
 
(54 intermediate revisions by the same user not shown)
Line 1: Line 1:
TensorFlow is the famous machine learning library by Google
TensorFlow is the famous machine learning library by Google


==Install==
===Install TF2===
See https://www.tensorflow.org/install/pip
Install tensorflow and [https://www.tensorflow.org/addons/overview tensorflow-addons]
<pre>
pip install tensorflow-addons
</pre>
;Notes
* Note that [https://anaconda.org/anaconda/tensorflow anaconda/tensorflow] does not always have the latest version.
* If you prefer, you can install only cuda and cudnn from conda:
** See [https://www.tensorflow.org/install/source#linux https://www.tensorflow.org/install/source#linux] for a list of compatible Cuda and Cudnn versions.
** <code>conda search cudatoolkit</code> to which versions of cuda available
** Download [https://developer.nvidia.com/cuDNN cudnn] and copy the binaries to the environment's <code>Library/bin/</code> directory.
===Install TF1===
The last official version of TensorFlow v1 is 1.15. This version does not work on RTX 3000+ (Ampere) GPUs. Your code will run but output bad results.<br>
If you need TensorFlow v1, see [https://github.com/NVIDIA/tensorflow nvidia-tensorflow].
<pre>
pip install nvidia-pyindex
pip install nvidia-tensorflow
</pre>


==Usage (TF2)==
==Usage (TF2)==
Here we'll cover usage using TensorFlow 2 which has eager execution.<br>
Here we'll cover usage using TensorFlow 2 which has eager execution.<br>
This is using the Keras API in tensorflow.keras.
This is using the Keras API in tensorflow.keras.
===Basics===
===Keras Pipeline===
===Training Loop===
[https://www.tensorflow.org/api_docs/python/tf/keras/Model tf.keras.Model]
 
The general pipeline using Keras is:
* Define a model, typically using [https://www.tensorflow.org/api_docs/python/tf/keras/Sequential tf.keras.Sequential]
* Call [https://www.tensorflow.org/api_docs/python/tf/keras/Model#compile <code>model.compile</code>]
** Here you pass in your optimizer, loss function, and metrics.
* Train your model by calling [https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit <code>model.fit</code>]
** Here you pass in your training data, batch size, number of epochs, and training callbacks
** For more information about callbacks, see [https://www.tensorflow.org/guide/keras/custom_callback Keras custom callbacks].
 
After training, you can use your model by calling [https://www.tensorflow.org/api_docs/python/tf/keras/Model#evaluate <code>model.evaluate</code>]
 
===Custom Models===
An alternative way to define a model is by extending the Model class:
 
* Write a python class which extends <code>tf.keras.Model</code>
* Implement a forward pass in the <code>call</code> method
 
See [https://www.tensorflow.org/guide/keras/custom_layers_and_models#building_models Tensorflow: Custom Layers And Models #Building Models]
 
===Custom Training Loop===
[https://www.tensorflow.org/guide/keras/train_and_evaluate#part_ii_writing_your_own_training_evaluation_loops_from_scratch Reference]<br>
[https://www.tensorflow.org/guide/keras/train_and_evaluate#part_ii_writing_your_own_training_evaluation_loops_from_scratch Reference]<br>
You can write your own training loop.
While you can train using <code>model.compile</code> and <code>model.fit</code>, using your own custom training loop is much more flexable and easier to understand.
You can write your own training loop by doing the following:
<syntaxhighlight lang="python">
import tensorflow as tf
from tensorflow import keras
 
my_model = keras.Sequential([
    keras.Input(shape=(400,)),
    keras.layers.Dense(400, activation='relu'),
    keras.layers.Dense(400, activation='relu'),
    keras.layers.Dense(2)
])
 
optimizer = keras.optimizers.SGD(learning_rate=1e-3)
 
training_loss = []
validation_loss = []
for epoch in range(100):
    print('Start of epoch %d' % (epoch,))
    for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):
        with tf.GradientTape() as tape:
            guess = my_model(x_batch_train)
            loss_value = my_custom_loss(y_batch_train, guess)
 
        # Use the gradient tape to automatically retrieve
        # the gradients of the trainable variables with respect to the loss.
        grads = tape.gradient(loss_value, my_model.trainable_weights)
 
        # Run one step of gradient descent by updating
        # the value of the variables to minimize the loss.
        optimizer.apply_gradients(zip(grads, my_model.trainable_weights))
 
        # Log every 200 batches.
        if step % 200 == 0:
            print('Training loss at step %s: %s' % (step, float(loss_value)))
        training_loss.append(loss_value)
        guess_validation = model(x_validation)
        validation_loss.append(my_custom_loss(y_validation, guess_validation))
</syntaxhighlight>


===Save and Load Models===
===Save and Load Models===
[https://www.tensorflow.org/tutorials/keras/save_and_load Reference]
[https://www.tensorflow.org/tutorials/keras/save_and_load Reference]
===Custom Layers===
Extend <code>tf.keras.layer.Layer</code>
{{ hidden | ReflectionPadding2D |
[https://stackoverflow.com/questions/50677544/reflection-padding-conv2d SO Source]
<syntaxhighlight lang="python">
class ReflectionPadding2D(Layer):
    def __init__(self, padding=(1, 1), **kwargs):
        self.padding = tuple(padding)
        self.input_spec = [InputSpec(ndim=4)]
        super(ReflectionPadding2D, self).__init__(**kwargs)
    def compute_output_shape(self, s):
        """ If you are using "channels_last" configuration"""
        return (s[0], s[1] + 2 * self.padding[0], s[2] + 2 * self.padding[1], s[3])
    def call(self, x, mask=None):
        w_pad,h_pad = self.padding
        return tf.pad(x, [[0,0], [h_pad,h_pad], [w_pad,w_pad], [0,0] ], 'REFLECT')
</syntaxhighlight>
}}
{{ hidden | BilinearUpsample |
<syntaxhighlight lang="python">
class BilinearUpsample(layers.Layer):
  def __init__(self):
    super().__init__()
    self.input_spec = [keras.layers.InputSpec(ndim=4)]
  def compute_output_shape(self, shape):
      return shape[0], 2 * shape[1], 2 * shape[2], shape[3]
  def call(self, inputs, training=None, mask=None):
    new_height = int(2 * inputs.shape[1])
    new_width = int(2 * inputs.shape[2])
    return tf.image.resize_images(inputs, [new_height, new_width])
</syntaxhighlight>
}}
==Operators==
===Matrix Multiplication===
The two matrix multiplication operators are:
* [https://www.tensorflow.org/api_docs/python/tf/linalg/matmul <code>tf.linalg.matmul</code>] (also aliased as <code>tf.matmul</code>)
* [https://www.tensorflow.org/api_docs/python/tf/linalg/matvec <code>tf.linalg.matvec</code>]
New: With both operators, the first \(k-2\) dimensions can now be the batch size. 
E.g. If \(A\) is \(b_1 \times b_2 \times 3 \times 3\) and \(B\) is \(b_1 \times b_2 \times 3\), you can multiply them with \(C = \operatorname{tf.linalg.matvec}(A,B)\) and \(C\) will be \(b_1 \times b_2 \times 3\). 
Also the batch size in A can be 1 and it will properly broadcast to the same size as \(B\).


==Usage (TF1)==
==Usage (TF1)==
In TF1, you first build a computational graph by chaining commands with placeholders and constant variables. 
Then, you execute the graph in a <code>tf.Session()</code>.
{{hidden | TF1 MNIST Example |
<syntaxhighlight lang="python">
import tensorflow as tf
from tensorflow import keras
import numpy as np
NUM_EPOCHS = 10
BATCH_SIZE = 64
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
rng = np.random.default_rng()
classification_model = keras.Sequential([
    keras.Input(shape=(28, 28, 1)),
    keras.layers.Conv2D(16, 3, padding="SAME"),
    keras.layers.ReLU(),
    keras.layers.Conv2D(16, 3, padding="SAME"),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10, activation='relu'),
])
x_in = tf.compat.v1.placeholder(dtype=tf.float32, shape=(None, 28, 28, 1))
logits = classification_model(x_in)
gt_classes = tf.compat.v1.placeholder(dtype=tf.int32, shape=(None,))
loss = tf.losses.softmax_cross_entropy(tf.one_hot(gt_classes, 10), logits)
optimizer = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(loss)
with tf.compat.v1.Session() as sess:
    sess.run(tf.compat.v1.global_variables_initializer())
    global_step = 0
    for epoch in range(NUM_EPOCHS):
        x_count = x_train.shape[0]
        image_ordering = rng.choice(range(x_count), x_count, replace=False)
        current_idx = 0
        while current_idx < x_count:
            my_indices = image_ordering[current_idx:min(current_idx + BATCH_SIZE, x_count)]
            x = x_train[my_indices]
            x = x[:, :, :, None] / 255
            logits_val, loss_val, _ = sess.run((logits, loss, optimizer), {
                x_in: x,
                gt_classes: y_train[my_indices]
            })
            if global_step % 100 == 0:
                print("Loss", loss_val)
            current_idx += BATCH_SIZE
            global_step += 1
</syntaxhighlight>
}}
===Batch Normalization===
See [https://www.tensorflow.org/api_docs/python/tf/compat/v1/layers/batch_normalization <code>tf.compat.v1.layers.batch_normalization</code>]
When training with batchnorm, you need to run <code>tf.GraphKeys.UPDATE_OPS</code> in your session to update the batchnorm variables or they will not be updated.
These variables do not contribute to the loss when training is true so they will not by updated by the optimizer.
<syntaxhighlight lang="python">
update_ops = tf.compat.v1.get_collection(tf.GraphKeys.UPDATE_OPS)
train_op = optimizer.minimize(loss)
train_op = tf.group([train_op, update_ops])
</syntaxhighlight>
==Estimators==
[https://towardsdatascience.com/first-contact-with-tensorflow-estimator-69a5e072998d First Contact w/ TF Estimator (TDS)]<br>
===Training Statistics===
[https://stackoverflow.com/questions/48940155/tensorflow-is-there-a-way-to-store-the-training-loss-in-tf-estimator Reference]<br>
You can extract the training loss from the events file in tensorflow.
==Tensorflow Addons==
<pre>
pip install tensorflow-addons
</pre>
===<code>tfa.image.interpolate_bilinear</code>===
[https://www.tensorflow.org/addons/api_docs/python/tfa/image/interpolate_bilinear Reference]
This is a bilinear interpolation. It is equivalent to PyTorch's grid_sample. 
However, you need to reshape the grid to a nx2 array and make sure <code>indexing='xy'</code> when calling the function. 
You can reshape the output back to the dimensions of your original image.
==Tensorflow Graphics==
<pre>
pip install tensorflow-graphics --upgrade
</pre>
You may need to install a static openexr from [https://www.lfd.uci.edu/~gohlke/pythonlibs/#openexr https://www.lfd.uci.edu/~gohlke/pythonlibs/#openexr].

Latest revision as of 14:23, 31 May 2023

TensorFlow is the famous machine learning library by Google

Install

Install TF2

See https://www.tensorflow.org/install/pip Install tensorflow and tensorflow-addons

pip install tensorflow-addons
Notes
  • Note that anaconda/tensorflow does not always have the latest version.
  • If you prefer, you can install only cuda and cudnn from conda:

Install TF1

The last official version of TensorFlow v1 is 1.15. This version does not work on RTX 3000+ (Ampere) GPUs. Your code will run but output bad results.
If you need TensorFlow v1, see nvidia-tensorflow.

pip install nvidia-pyindex
pip install nvidia-tensorflow

Usage (TF2)

Here we'll cover usage using TensorFlow 2 which has eager execution.
This is using the Keras API in tensorflow.keras.

Keras Pipeline

tf.keras.Model

The general pipeline using Keras is:

  • Define a model, typically using tf.keras.Sequential
  • Call model.compile
    • Here you pass in your optimizer, loss function, and metrics.
  • Train your model by calling model.fit
    • Here you pass in your training data, batch size, number of epochs, and training callbacks
    • For more information about callbacks, see Keras custom callbacks.

After training, you can use your model by calling model.evaluate

Custom Models

An alternative way to define a model is by extending the Model class:

  • Write a python class which extends tf.keras.Model
  • Implement a forward pass in the call method

See Tensorflow: Custom Layers And Models #Building Models

Custom Training Loop

Reference
While you can train using model.compile and model.fit, using your own custom training loop is much more flexable and easier to understand. You can write your own training loop by doing the following:

import tensorflow as tf
from tensorflow import keras

my_model = keras.Sequential([
    keras.Input(shape=(400,)),
    keras.layers.Dense(400, activation='relu'),
    keras.layers.Dense(400, activation='relu'),
    keras.layers.Dense(2)
])

optimizer = keras.optimizers.SGD(learning_rate=1e-3)

training_loss = []
validation_loss = []
for epoch in range(100):
    print('Start of epoch %d' % (epoch,))
    for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):
        with tf.GradientTape() as tape:
            guess = my_model(x_batch_train)
            loss_value = my_custom_loss(y_batch_train, guess)

        # Use the gradient tape to automatically retrieve
        # the gradients of the trainable variables with respect to the loss.
        grads = tape.gradient(loss_value, my_model.trainable_weights)

        # Run one step of gradient descent by updating
        # the value of the variables to minimize the loss.
        optimizer.apply_gradients(zip(grads, my_model.trainable_weights))

        # Log every 200 batches.
        if step % 200 == 0:
            print('Training loss at step %s: %s' % (step, float(loss_value)))
        training_loss.append(loss_value)
        guess_validation = model(x_validation)
        validation_loss.append(my_custom_loss(y_validation, guess_validation))

Save and Load Models

Reference

Custom Layers

Extend tf.keras.layer.Layer

ReflectionPadding2D

SO Source

class ReflectionPadding2D(Layer):
    def __init__(self, padding=(1, 1), **kwargs):
        self.padding = tuple(padding)
        self.input_spec = [InputSpec(ndim=4)]
        super(ReflectionPadding2D, self).__init__(**kwargs)

    def compute_output_shape(self, s):
        """ If you are using "channels_last" configuration"""
        return (s[0], s[1] + 2 * self.padding[0], s[2] + 2 * self.padding[1], s[3])

    def call(self, x, mask=None):
        w_pad,h_pad = self.padding
        return tf.pad(x, [[0,0], [h_pad,h_pad], [w_pad,w_pad], [0,0] ], 'REFLECT')
BilinearUpsample
class BilinearUpsample(layers.Layer):

  def __init__(self):
    super().__init__()
    self.input_spec = [keras.layers.InputSpec(ndim=4)]

  def compute_output_shape(self, shape):
      return shape[0], 2 * shape[1], 2 * shape[2], shape[3]

  def call(self, inputs, training=None, mask=None):
    new_height = int(2 * inputs.shape[1])
    new_width = int(2 * inputs.shape[2])
    return tf.image.resize_images(inputs, [new_height, new_width])

Operators

Matrix Multiplication

The two matrix multiplication operators are:

New: With both operators, the first \(k-2\) dimensions can now be the batch size.
E.g. If \(A\) is \(b_1 \times b_2 \times 3 \times 3\) and \(B\) is \(b_1 \times b_2 \times 3\), you can multiply them with \(C = \operatorname{tf.linalg.matvec}(A,B)\) and \(C\) will be \(b_1 \times b_2 \times 3\).
Also the batch size in A can be 1 and it will properly broadcast to the same size as \(B\).

Usage (TF1)

In TF1, you first build a computational graph by chaining commands with placeholders and constant variables.
Then, you execute the graph in a tf.Session().

TF1 MNIST Example
import tensorflow as tf
from tensorflow import keras
import numpy as np

NUM_EPOCHS = 10
BATCH_SIZE = 64

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
rng = np.random.default_rng()

classification_model = keras.Sequential([
    keras.Input(shape=(28, 28, 1)),
    keras.layers.Conv2D(16, 3, padding="SAME"),
    keras.layers.ReLU(),
    keras.layers.Conv2D(16, 3, padding="SAME"),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10, activation='relu'),
])
x_in = tf.compat.v1.placeholder(dtype=tf.float32, shape=(None, 28, 28, 1))
logits = classification_model(x_in)
gt_classes = tf.compat.v1.placeholder(dtype=tf.int32, shape=(None,))
loss = tf.losses.softmax_cross_entropy(tf.one_hot(gt_classes, 10), logits)
optimizer = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(loss)

with tf.compat.v1.Session() as sess:
    sess.run(tf.compat.v1.global_variables_initializer())
    global_step = 0
    for epoch in range(NUM_EPOCHS):
        x_count = x_train.shape[0]
        image_ordering = rng.choice(range(x_count), x_count, replace=False)
        current_idx = 0
        while current_idx < x_count:
            my_indices = image_ordering[current_idx:min(current_idx + BATCH_SIZE, x_count)]
            x = x_train[my_indices]
            x = x[:, :, :, None] / 255
            logits_val, loss_val, _ = sess.run((logits, loss, optimizer), {
                x_in: x,
                gt_classes: y_train[my_indices]
            })
            if global_step % 100 == 0:
                print("Loss", loss_val)

            current_idx += BATCH_SIZE
            global_step += 1

Batch Normalization

See tf.compat.v1.layers.batch_normalization When training with batchnorm, you need to run tf.GraphKeys.UPDATE_OPS in your session to update the batchnorm variables or they will not be updated. These variables do not contribute to the loss when training is true so they will not by updated by the optimizer.

update_ops = tf.compat.v1.get_collection(tf.GraphKeys.UPDATE_OPS)
train_op = optimizer.minimize(loss)
train_op = tf.group([train_op, update_ops])

Estimators

First Contact w/ TF Estimator (TDS)

Training Statistics

Reference
You can extract the training loss from the events file in tensorflow.


Tensorflow Addons

pip install tensorflow-addons

tfa.image.interpolate_bilinear

Reference

This is a bilinear interpolation. It is equivalent to PyTorch's grid_sample.
However, you need to reshape the grid to a nx2 array and make sure indexing='xy' when calling the function.
You can reshape the output back to the dimensions of your original image.

Tensorflow Graphics

pip install tensorflow-graphics --upgrade

You may need to install a static openexr from https://www.lfd.uci.edu/~gohlke/pythonlibs/#openexr.