10_Introduction to Artificial Neural Networks w Keras_2_tensorflow2.1_Anaconda3-2019.10(python3.7.4)

10_Introduction to Artificial Neural Networks w Keras0_HuberLoss_astype_dtype_DNN_MLP_G.gv.pdf_mnist

https://blog.youkuaiyun.com/Linli522362242/article/details/106433059

Using dense() instead of neuron_layer()

Note: previous releases used tensorflow.contrib.layers.fully_connected() rather than tf.layers.dense()(which did not exist when this chapter was written). It is now preferable to use tf.layers.dense(), because anything in the contrib module may change or be deleted without notice. The dense() function is almost identical to the fully_connected() function, except for a few minor differences:

• several parameters are renamed: scope becomes name, activation_fn becomes activation (and similarly the _fn suffix is removed from other parameters such as normalizer_fn), weights_initializer becomes kernel_initializer, etc.
• the default activation is now None rather than tf.nn.relu.
• a few more differences are presented in chapter 11.

Construction Phase https://blog.youkuaiyun.com/Linli522362242/article/details/106433059

Let's start. First we need to import the tensorflow library. Then we must specify the number of inputs and outputs, and set the number of hidden neurons in each layer:

import tensorflow as tf

n_inputs = 28*28 #MNIST
n_hidden1 = 300
n_hidden2 = 100
n_outputs =10

tf.reset_default_graph()

X = tf.placeholder( tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder( tf.int32, shape=(None), name="y")

 Using dense() instead of neuron_layer()

# #n_inputs = 28*28+1 ==> n_hidden1 = 300 ==> n_hidden2 = 100 ==> n_outputs = 10
# with tf.name_scope("dnn"):
#     hidden1 = neuron_layer( X, n_hidden1, name="hidden1", activation=tf.nn.relu )
#     hidden2 = neuron_layer( hidden1, n_hidden2, name="hidden2", activation=tf.nn.relu )
#     logits = neuron_layer( hidden2, n_outputs, name="outputs" )

with tf.name_scope("dnn"): #differ from fully_connected() function
    hidden1 = tf.layers.dense( X, n_hidden1, name="hidden1", activation=tf.nn.relu)
    hidden2 = tf.layers.dense( hidden1, n_hidden2, name="hidden2", activation=tf.nn.relu)
    logits = tf.layers.dense(hidden2, n_outputs, name="outputs")

with tf.name_scope("loss"):
    xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
    loss = tf.reduce_mean( xentropy, name="loss")

with tf.name_scope("loss"):
    xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
    loss = tf.reduce_mean( xentropy, name="loss")

with tf.name_scope("eval"):
    #k=1 is the top-one class  # tf.nn.in_top_k(predictions, targets, k, name=None)
    correct = tf.nn.in_top_k(logits, y, 1) #logits.shape : (samples, class_dimensions
    accuracy = tf.reduce_mean( tf.cast(correct, tf.float32) )

init = tf.global_variables_initializer()
saver = tf.train.Saver()

Execution Phase

import keras
 
(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()
X_train = X_train.astype(np.float32).reshape(-1, 28*28) / 255.0
X_test = X_test.astype(np.float32).reshape(-1, 28*28) / 255.0
y_train = y_train.astype(np.int32)
y_test = y_test.astype(np.int32)
X_valid, X_train = X_train[:5000], X_train[5000:]
y_valid, y_train = y_train[:5000], y_train[5000:]


def shuffle_batch( X, y, batch_size ):
    rnd_idx = np.random.permutation( len(X) )
    n_batches = len(X) // batch_size
    # np.array_split(rnd_idx, n_batches) # split the list of rnd_idx to n_batches groups
    for batch_idx in np.array_split(rnd_idx, n_batches):
        X_batch, y_batch = X[batch_idx], y[batch_idx]
        yield X_batch, y_batch

n_epochs = 20 ########
n_batches = 50

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
            sess.run( training_op, feed_dict={X: X_batch, y: y_batch} )
        acc_batch = accuracy.eval( feed_dict={X: X_batch, y: y_batch} )
        acc_val = accuracy.eval( feed_dict = {X: X_valid, y: y_valid} )
        print( epoch, "(Last) Batch accuracy: ", acc_batch, "~ Val accuracy: ", acc_val )
        
    save_path = saver.save(sess, "./my_model_final.ckpt")

import tfgraphviz as tfg
tfg.board( tf.get_default_graph() ).view()

'G.gv.pdf'  https://blog.youkuaiyun.com/Linli522362242/article/details/106433059
VS neuron_layer

Implementing MLPs with Keras

     Keras is a high-level Deep Learning API that allows you to easily build, train, evaluate, and execute all sorts of neural networks. Its documentation (or specification) is available at https://keras.io/. The reference implementation, also called Keras, was developed by François Chollet as part of a research project13 and was released as an open source project in March 2015. It quickly gained popularity, owing to its ease of
use, flexibility, and beautiful design. To perform the heavy computations required by neural networks, this reference implementation relies on a computation backend. At present, you can choose from three popular open source Deep Learning libraries: TensorFlow, Microsoft Cognitive Toolkit (CNTK), and Theano. Therefore, to avoid any confusion, we will refer to this reference implementation as multibackend Keras.

     Since late 2016, other implementations have been released. You can now run Keras on Apache MXNet, Apple’s Core ML, JavaScript or TypeScript (to run Keras code in a web browser), and PlaidML (which can run on all sorts of GPU devices, not just Nvidia). Moreover, TensorFlow itself now comes bundled with its own Keras implementation, tf.keras. It only supports TensorFlow as the backend, but it has the advantage of offering some very useful extra features (see Figure 10-10): for example, it supports TensorFlow’s Data API, which makes it easy to load and preprocess data efficiently. For this reason, we will use tf.keras in this book. However, in this chapter we will not use any of the TensorFlow-specific features, so the code should run fine on other Keras implementations as well (at least in Python), with only minor modifications, such as changing the imports.

Figure 10-10. Two implementations of the Keras API: multibackend Keras (left) and tf.keras (right)

     The most popular Deep Learning library, after Keras and TensorFlow, is Facebook’s PyTorch library. The good news is that its API is quite similar to Keras’s (in part because both APIs were inspired by Scikit-Learn and Chainer), so once you know Keras, it is not difficult to switch to PyTorch, if you ever want to. PyTorch’s popularity grew exponentially in 2018, largely thanks to its simplicity and excellent documentation, which were not TensorFlow 1.x’s main strengths. However, TensorFlow 2 is arguably just as simple as PyTorch, as it has adopted Keras as its official high-level API and its developers have greatly simplified and cleaned up the rest of the API. The documentation has also been completely reorganized, and it is much easier to find what you need now. Similarly, PyTorch’s main weaknesses (e.g., limited portability and no computation graph analysis) have been largely addressed in PyTorch 1.0. Healthy competition is beneficial to everyone.

All right, it’s time to code! As tf.keras is bundled with TensorFlow, let’s start by installing TensorFlow.

Install tensorflow 2

conda create --name tensorflow python=3.7.4

conda activate tensorflow  

conda info --envs 

python

exit()

conda list

conda install tensorflow

In jupyter notebook

conda install jupyter notebook

conda install Spyder
 

 pip install graphviz

Then, go to website : https://graphviz.gitlab.io/_pages/Download/Download_windows.html

to download .msi file, and install it and write down the directory where you install it 

install "for everyone"

# next step, we have to append it to system environment path

pip install pydot OR conda install pydot

import pydot
edg = [(1,2), (1,3), (1,4) , (3,4)]
g=pydot.graph_from_edges(edg)
g.write_jpeg('graph.jpg', prog = 'dot')

 

pip install tfgraphviz

conda install Pandas

conda install scikit-learn

conda install ipython

conda install matplotlib
... ...

 

import tensorflow as tf
from tensorflow import keras

tf.__version__

keras.__version__

     The second version is the version of the Keras API implemented by tf.keras. Note that it ends with -tf, highlighting the fact that tf.keras implements the Keras API, plus some extra TensorFlow-specific features.

     Now let’s use tf.keras! We’ll start by building a simple image classifier.

Building an Image Classifier Using the Sequential API

https://blog.youkuaiyun.com/Linli522362242/article/details/106562190