TensorFlow: Difference between revisions

(16 intermediate revisions by the same user not shown)

Line 2:

==Install==

* Install CUDA and CuDNN

* Create a conda environment with python 3.5+

** <code>conda create -n my_env python=3.8</code>

* Install with pip

===Install TF2===

See https://www.tensorflow.org/install/pip

Install tensorflow and [https://www.tensorflow.org/addons/overview tensorflow-addons]

<pre>

~~conda install tensorflow-gpu~~

pip install tensorflow-addons

</pre>

;Notes

* Note that [https://anaconda.org/anaconda/tensorflow anaconda/tensorflow] does not always have the latest version.

* If you prefer, you can install only cuda and cudnn from conda:

** See [https://www.tensorflow.org/install/source#linux https://www.tensorflow.org/install/source#linux] for a list of compatible Cuda and Cudnn versions.

Line 20:

Line 18:

===Install TF1===

~~Just install using Conda~~. ~~It's the easiest way~~.

The last official version of TensorFlow v1 is 1.15. This version does not work on RTX 3000+ (Ampere) GPUs. Your code will run but output bad results.<br>

If you need TensorFlow v1, see [https://github.com/NVIDIA/tensorflow nvidia-tensorflow].

<pre>

~~conda~~ install tensorflow~~-gpu=1.15~~

pip install nvidia-pyindex

pip install nvidia-tensorflow

</pre>

~~;Notes~~

* Conda will automatically install a compatible cuda and cudnn into the cuda environment. Your host OS only needs to have a sufficiently new version of nvidia drivers installed.

* Sometimes, I get <code>CUDNN_STATUS_INTERNAL_ERROR</code>. This is fixed by setting the environment variable <code>TF_FORCE_GPU_ALLOW_GROWTH=true</code> in my conda env. See [https://stackoverflow.com/questions/46826497/conda-set-ld-library-path-for-env-only Add env variables to conda env]

==Usage (TF2)==

Here we'll cover usage using TensorFlow 2 which has eager execution.<br>

This is using the Keras API in tensorflow.keras.

===~~Basics~~===

===Keras Pipeline===

[https://www.tensorflow.org/api_docs/python/tf/keras/Model tf.keras.Model]

The general pipeline using Keras is:

* Define a model, typically using [https://www.tensorflow.org/api_docs/python/tf/keras/Sequential tf.keras.Sequential]

* Call [https://www.tensorflow.org/api_docs/python/tf/keras/Model#compile <code>model.compile</code>]

** Here you pass in your optimizer, loss function, and metrics.

* Train your model by calling <code>model.fit</code>

* Train your model by calling [https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit <code>model.fit</code>]

** Here you pass in your training data, batch size, number of epochs, and training callbacks

** For more information about callbacks, see [https://www.tensorflow.org/guide/keras/custom_callback Keras custom callbacks].

After training, you can use your model by calling <code>model.evaluate</code>

After training, you can use your model by calling [https://www.tensorflow.org/api_docs/python/tf/keras/Model#evaluate <code>model.evaluate</code>]

===Custom Models===

Line 61:

Line 58:

my_model = keras.Sequential([

keras.~~layers.Dense~~(~~400, input_shape~~=~~400, activation='relu'),~~

keras.Input(shape=(400,)),

~~keras.layers.Dense~~(400, ~~activation='relu'~~),

~~keras.layers.Dense(400, activation='relu'~~),

keras.layers.Dense(400, activation='relu'),

Line 150:

Line 145:

==Usage (TF1)==

In TF1, you first build a computational graph by chaining commands with ~~placeholder~~.

In TF1, you first build a computational graph by chaining commands with placeholders and constant variables.

Then, you execute the graph in a tf ~~session~~.

Then, you execute the graph in a <code>tf.Session()</code>.

{{hidden | TF1 MNIST Example |

import tensorflow as tf

from tensorflow import keras

import numpy as np

NUM_EPOCHS = 10

BATCH_SIZE = 64

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

rng = np.random.default_rng()

classification_model = keras.Sequential([

keras.Input(shape=(28, 28, 1)),

keras.layers.Conv2D(16, 3, padding="SAME"),

keras.layers.ReLU(),

keras.layers.Conv2D(16, 3, padding="SAME"),

keras.layers.ReLU(),

keras.layers.Flatten(),

keras.layers.Dense(10, activation='relu'),

])

x_in = tf.compat.v1.placeholder(dtype=tf.float32, shape=(None, 28, 28, 1))

logits = classification_model(x_in)

gt_classes = tf.compat.v1.placeholder(dtype=tf.int32, shape=(None,))

loss = tf.losses.softmax_cross_entropy(tf.one_hot(gt_classes, 10), logits)

optimizer = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(loss)

with tf.compat.v1.Session() as sess:

sess.run(tf.compat.v1.global_variables_initializer())

global_step = 0

for epoch in range(NUM_EPOCHS):

x_count = x_train.shape[0]

image_ordering = rng.choice(range(x_count), x_count, replace=False)

current_idx = 0

while current_idx < x_count:

my_indices = image_ordering[current_idx:min(current_idx + BATCH_SIZE, x_count)]

x = x_train[my_indices]

x = x[:, :, :, None] / 255

logits_val, loss_val, _ = sess.run((logits, loss, optimizer), {

x_in: x,

gt_classes: y_train[my_indices]

})

if global_step % 100 == 0:

print("Loss", loss_val)

current_idx += BATCH_SIZE

global_step += 1

</syntaxhighlight>

}}

===Batch Normalization===

See [https://www.tensorflow.org/api_docs/python/tf/compat/v1/layers/batch_normalization <code>tf.compat.v1.layers.batch_normalization</code>]

When training with batchnorm, you need to run <code>tf.GraphKeys.UPDATE_OPS</code> in your session to update the batchnorm variables or they will not be updated.

These variables do not contribute to the loss when training is true so they will not by updated by the optimizer.

update_ops = tf.compat.v1.get_collection(tf.GraphKeys.UPDATE_OPS)

train_op = optimizer.minimize(loss)

train_op = tf.group([train_op, update_ops])

</syntaxhighlight>

@@ Line 2: / Line 2: @@
 ==Install==
-* Install CUDA and CuDNN
-* Create a conda environment with python 3.5+
-** <code>conda create -n my_env python=3.8</code>
-* Install with pip
 ===Install TF2===
+See https://www.tensorflow.org/install/pip
 Install tensorflow and [https://www.tensorflow.org/addons/overview tensorflow-addons]
 <pre>
-conda install tensorflow-gpu
 pip install tensorflow-addons
 </pre>
+;Notes
+* Note that [https://anaconda.org/anaconda/tensorflow anaconda/tensorflow] does not always have the latest version.
 * If you prefer, you can install only cuda and cudnn from conda:
 ** See [https://www.tensorflow.org/install/source#linux https://www.tensorflow.org/install/source#linux] for a list of compatible Cuda and Cudnn versions.
@@ Line 20: / Line 18: @@
 ===Install TF1===
-Just install using Conda. It's the easiest way.
+The last official version of TensorFlow v1 is 1.15. This version does not work on RTX 3000+ (Ampere) GPUs. Your code will run but output bad results.<br>
+If you need TensorFlow v1, see [https://github.com/NVIDIA/tensorflow nvidia-tensorflow].
 <pre>
-conda install tensorflow-gpu=1.15
+pip install nvidia-pyindex
+pip install nvidia-tensorflow
 </pre>
-;Notes
-* Conda will automatically install a compatible cuda and cudnn into the cuda environment. Your host OS only needs to have a sufficiently new version of nvidia drivers installed.
-* Sometimes, I get <code>CUDNN_STATUS_INTERNAL_ERROR</code>. This is fixed by setting the environment variable <code>TF_FORCE_GPU_ALLOW_GROWTH=true</code> in my conda env. See [https://stackoverflow.com/questions/46826497/conda-set-ld-library-path-for-env-only Add env variables to conda env]
 ==Usage (TF2)==
 Here we'll cover usage using TensorFlow 2 which has eager execution.<br>
 This is using the Keras API in tensorflow.keras.
-===Basics===
+===Keras Pipeline===
+[https://www.tensorflow.org/api_docs/python/tf/keras/Model tf.keras.Model]
 The general pipeline using Keras is:
 * Define a model, typically using [https://www.tensorflow.org/api_docs/python/tf/keras/Sequential tf.keras.Sequential]
 * Call [https://www.tensorflow.org/api_docs/python/tf/keras/Model#compile <code>model.compile</code>]
 ** Here you pass in your optimizer, loss function, and metrics.
-* Train your model by calling <code>model.fit</code>
+* Train your model by calling [https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit <code>model.fit</code>]
 ** Here you pass in your training data, batch size, number of epochs, and training callbacks
 ** For more information about callbacks, see [https://www.tensorflow.org/guide/keras/custom_callback Keras custom callbacks].
-After training, you can use your model by calling <code>model.evaluate</code>
+After training, you can use your model by calling [https://www.tensorflow.org/api_docs/python/tf/keras/Model#evaluate <code>model.evaluate</code>]
 ===Custom Models===
@@ Line 61: / Line 58: @@
 my_model = keras.Sequential([
-     keras.layers.Dense(400, input_shape=400, activation='relu'),
+     keras.Input(shape=(400,)),
-    keras.layers.Dense(400, activation='relu'),
-    keras.layers.Dense(400, activation='relu'),
      keras.layers.Dense(400, activation='relu'),
      keras.layers.Dense(400, activation='relu'),
@@ Line 150: / Line 145: @@
 ==Usage (TF1)==
-In TF1, you first build a computational graph by chaining commands with placeholder.
+In TF1, you first build a computational graph by chaining commands with placeholders and constant variables.
-Then, you execute the graph in a tf session.
+Then, you execute the graph in a <code>tf.Session()</code>.
+{{hidden | TF1 MNIST Example |
 <syntaxhighlight lang="python">
 import tensorflow as tf
+from tensorflow import keras
+import numpy as np
+NUM_EPOCHS = 10
+BATCH_SIZE = 64
+(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
+rng = np.random.default_rng()
+classification_model = keras.Sequential([
+    keras.Input(shape=(28, 28, 1)),
+    keras.layers.Conv2D(16, 3, padding="SAME"),
+    keras.layers.ReLU(),
+    keras.layers.Conv2D(16, 3, padding="SAME"),
+    keras.layers.ReLU(),
+    keras.layers.Flatten(),
+    keras.layers.Dense(10, activation='relu'),
+])
+x_in = tf.compat.v1.placeholder(dtype=tf.float32, shape=(None, 28, 28, 1))
+logits = classification_model(x_in)
+gt_classes = tf.compat.v1.placeholder(dtype=tf.int32, shape=(None,))
+loss = tf.losses.softmax_cross_entropy(tf.one_hot(gt_classes, 10), logits)
+optimizer = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(loss)
+with tf.compat.v1.Session() as sess:
+    sess.run(tf.compat.v1.global_variables_initializer())
+    global_step = 0
+    for epoch in range(NUM_EPOCHS):
+        x_count = x_train.shape[0]
+        image_ordering = rng.choice(range(x_count), x_count, replace=False)
+        current_idx = 0
+        while current_idx < x_count:
+            my_indices = image_ordering[current_idx:min(current_idx + BATCH_SIZE, x_count)]
+            x = x_train[my_indices]
+            x = x[:, :, :, None] / 255
+            logits_val, loss_val, _ = sess.run((logits, loss, optimizer), {
+                x_in: x,
+                gt_classes: y_train[my_indices]
+            })
+            if global_step % 100 == 0:
+                print("Loss", loss_val)
+            current_idx += BATCH_SIZE
+            global_step += 1
+</syntaxhighlight>
+}}
+===Batch Normalization===
+See [https://www.tensorflow.org/api_docs/python/tf/compat/v1/layers/batch_normalization <code>tf.compat.v1.layers.batch_normalization</code>]
+When training with batchnorm, you need to run <code>tf.GraphKeys.UPDATE_OPS</code> in your session to update the batchnorm variables or they will not be updated.
+These variables do not contribute to the loss when training is true so they will not by updated by the optimizer.
+<syntaxhighlight lang="python">
+update_ops = tf.compat.v1.get_collection(tf.GraphKeys.UPDATE_OPS)
+train_op = optimizer.minimize(loss)
+train_op = tf.group([train_op, update_ops])
 </syntaxhighlight>