多层full-connect 神经网络测试

1、搭建了一个多层full-connect神经网络，用于mnist识别

2、代码中测试了几种cross_entropy的用法，并进行对比，cross_entropy1，cross_entropy2，cross_entropy3对输入参数各有要求，所得到的结果是相同的。

3、在测试过程中出现了loss=NaN的情况，上网搜索原因后发现是两个原因导致的，1）学习率过高，于是将learning_rate调整为0.001 。 2）cross_entropy中出现了log(0)的情况，所以用tf.clip_by_value() 对log的输入值进行来限制

4、发现多次测试，不同的变量初始化，最终的准确率存在0.1级别的差距

5、用SGD优化算法，测试了多种loss函数，包括loss=loss1_mean,loss=loss1_sum,loss1_mean=cross_entropy1,使用loss=loss1_mean效果最好，基本每次都能收敛，另外两个存在不收敛情况（有时发散，有时收敛，不稳定）。

6、疑问：loss过大不好？还是因为梯度下降法本身存在缺陷，时常会收敛到局部极小值，而不是全局最小值。又测试了

AdamOptimizer优化器，对多个损失函数都比较鲁棒，都能收敛，所以应该是梯度下降法本身存在的问题。

 

#-*-coding:utf-8-*-
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
LEARNING_RATE = 0.001
BATCH_SIZE=100

##get data
mnist = input_data.read_data_sets('./MNIST_data',one_hot=True)

## data_input
x = tf.placeholder(tf.float32,[None,784])
y_ = tf.placeholder(tf.float32,[None,10])

##forward process

W1 = tf.Variable(tf.truncated_normal([784, 1024], stddev=0.1))
b1 = tf.Variable(tf.zeros([1024]))

W2 = tf.Variable(tf.truncated_normal([1024, 800], stddev=0.1))
b2 = tf.Variable(tf.zeros([800]))

W3 = tf.Variable(tf.truncated_normal([800,500],stddev=0.1))
b3 = tf.Variable(tf.zeros([500]))

W4 = tf.Variable(tf.truncated_normal([500,10],stddev=0.1))
b4 = tf.Variable(tf.zeros([10]))


layer1 = tf.nn.relu(tf.matmul(x, W1) + b1)
layer2 = tf.nn.relu(tf.matmul(layer1,W2)+b2)
layer3 = tf.nn.relu(tf.matmul(layer2,W3)+b3)
layer4 = tf.nn.relu(tf.matmul(layer3,W4)+b4)

y = layer4
y1 = tf.nn.softmax(y)
y1 = tf.clip_by_value(y1,1e-10,1e+100)

##loss
cross_entropy1 = -tf.reduce_sum(y_*tf.log(y1),1)
cross_entropy2 = tf.nn.softmax_cross_entropy_with_logits(labels=y_,logits=y)
cross_entropy3 = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=tf.argmax(y_,1),logits=y)

loss1_mean = tf.reduce_mean(cross_entropy1)
loss1_sum = tf.reduce_sum(cross_entropy1)

loss2_mean = tf.reduce_mean(cross_entropy2)
loss2_sum = tf.reduce_sum(cross_entropy2)

loss3_mean = tf.reduce_mean(cross_entropy3)
loss3_sum = tf.reduce_sum(cross_entropy3)

###########

loss = loss1_mean

train_step = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(loss)

correct_predict = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(y_,1),tf.argmax(y,1)),tf.float32))

##train
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    validata_x = mnist.validation.images
    validata_y = mnist.validation.labels

    for i in range(30000):
        xs,ys = mnist.train.next_batch(batch_size=BATCH_SIZE)
        sess.run(train_step,feed_dict={
            x:xs,y_:ys})

        if i%1000==0:
            correct_value = sess.run(correct_predict,feed_dict={x:xs,y_:ys})
            print("step %d,correct_prediction = %.5f"%(i,correct_value))


            ####test cross_entropy
            # print('cross_entropy1:\n',sess.run(cross_entropy1, feed_dict={x: xs, y_: ys}))
            # print('cross_entropy2:\n',sess.run(cross_entropy2, feed_dict={x: xs, y_: ys}))
            # print('cross_entropy3:\n',sess.run(cross_entropy3, feed_dict={x: xs, y_: ys}))
            #
            # #
            # print('loss1_mean:\n', sess.run(loss1_mean, feed_dict={x: xs, y_: ys}))
            # print('loss1_sum:\n', sess.run(loss1_sum, feed_dict={x: xs, y_: ys}))
            # print('\n')