纠错： 深度学习模型优化时快速收敛

最近在做问答系统，用CNN深度学习模型分别对问题、正答案和负答案提取特征，得到各自的特征向量。正答案与问题之间的距离比负答案与问题之间的距离要近，距离用夹角的cos值来表示，目标函数是由此确定的。模型出自《Applying Deep Learning To Answer Selection: A Study And An Open Task》，如图1所示。

图 1
用tensorflow架构实现该模型，然而做模型优化时，5步以内模型的loss就为0了，精度达到了1，如图2所示。‘

图 2
迷糊了一个星期，才找到问题点。
原来是因为在用CNN做特征提取时，我对三个输入（问题、正答案、负答案）分别做了模型weight初始化，相当于建立了三个CNN分别对问题、正答案、负答案做特征抽取。代码如下，调用了conv函数三遍，初始化了W三次，建立了三个CNN模型。

# define a convolution function 
    def conv(input_data):
        pooled_outputs = []
        for i, filter_size in enumerate(filter_sizes):
                with tf.name_scope("conv-maxpool-%s" % filter_size):
                    # Convolution Layer         
            filter_shape = [filter_size, embedding_size, 1, num_filters]
                W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
                b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
                conv = tf.nn.conv2d(input_data, W, strides=[1, 1, 1, 1],
                padding="VALID", name="conv")
            h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")    
                pooled = tf.nn.max_pool(
                h,
                        ksize=[1, sequence_length - filter_size + 1, 1, 1],
                        strides=[1, 1, 1, 1],
                        padding='VALID',
                        name="pool")    # shape of pooled is [batch_size,1,1,num_filters]
                pooled_outputs.append(pooled)
        return pooled_outputs
    pooled_outputs1=conv(self.embedded_chars1_expanded) # conv-pool outputs
    pooled_outputs2=conv(self.embedded_chars2_expanded)
    pooled_outputs3=conv(self.embedded_chars3_expanded) 

为什么用三个CNN模型做QA系统的答案选择不行呢？此时的模型相当于如下图3的结构。

图 3
三个CNN模型分别做特征提取，导致模型对负答案失去约束力，也就是说负答案的卷积神经网络模型使它始终离问题很远（夹角的cos值小），即使此负答案CNN模型的输入是正答案，距离仍然会远（夹角仍然会小）。打个形象的比喻，就像是让负答案乘以0，无论输入怎么变，输出都是零，这样的约束没有了意义。
对代码进行如下修改。

pooled_outputs1 = []
        pooled_outputs2 = []
        pooled_outputs3 = []
        for i, filter_size in enumerate(filter_sizes):
            with tf.name_scope("conv-maxpool-%s" % filter_size):
                filter_shape = [filter_size, embedding_size, 1, num_filters]
                W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
                b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
                conv = tf.nn.conv2d(self.embedded_chars1_expanded, W, strides=[1, 1, 1, 1],
                                    padding="VALID", name="conv")
                h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
                pooled = tf.nn.max_pool(
                                h,
                                ksize=[1, sequence_length - filter_size + 1, 1, 1],
                                strides=[1, 1, 1, 1],
                                padding='VALID',
                                name="pool")  # shape of pooled is [batch_size,1,1,num_filters]
                pooled_outputs1.append(pooled)
                conv = tf.nn.conv2d(self.embedded_chars2_expanded, W, strides=[1, 1, 1, 1],
                                    padding="VALID", name="conv")
                h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
                        pooled = tf.nn.max_pool(
                                h,
                                ksize=[1, sequence_length - filter_size + 1, 1, 1],
                                strides=[1, 1, 1, 1],
                                padding='VALID',
                                name="pool")  # shape of pooled is [batch_size,1,1,num_filters]
                        pooled_outputs2.append(pooled)
                        conv = tf.nn.conv2d(self.embedded_chars3_expanded, W, strides=[1, 1, 1, 1],
                                    padding="VALID", name="conv")
        # print('\n--- shape of cov is {}'.format(conv.get_shape()))
        # Apply nonlinearity
                        h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
        # Max-pooling over the outputs
                        pooled = tf.nn.max_pool(
                                h,
                                ksize=[1, sequence_length - filter_size + 1, 1, 1],
                                strides=[1, 1, 1, 1],
                                padding='VALID',
                                name="pool")  # shape of pooled is [batch_size,1,1,num_filters]
                        pooled_outputs3.append(pooled)

改过的程序可以正常运行了，准确率和论文描述的相当。