Maxout Networks

$maxout$ 的论文，参考 Goodfellow et al., 2013a
上述论文的基本翻译，参考知乎回答

本质上讲，$maxout$ 的核心在于：

其中：

其中，$j$ 为$k$ 的$cursor$ ，$i$ 为 $hidden$ $layer$ 的 $cursor$ , 表示$z_{ij}$ 由输入张量$x$ 与第$j$ 个$W$矩阵的第$i$项相称，在$k$个相乘的结果中取最大值为隐藏层$h_{i}$的值。

其他计算$hidden\ layer$ 的方法：

即直接$x$ 与 $W$ 相乘
对上图所示的神经元的$maxout$的计算方法：

$x$ 与多个 $W$ 相乘，取最大值得到$z_{ij}$
值得注意的是，此处为$W_{i}$，即为某一向量，而非需要多个$W$矩阵

因而在实践中，常常将$W$ 的规模乘以$k$ 倍（需要手动设置）， 得到$k$ 倍的$z_{ij}$ 后， 每$k$ 个选出其中最大的，作为对应的$h_{i}$的值。这一行为和max-pooling 表现一致

$tensorflow$ 实现：
基于$max-pooling$

import numpy as np
import tensorflow as tf

"""
Maxout OP from https://arxiv.org/abs/1302.4389
Max pooling is performed in given filter/channel dimension. This can also be
used after fully-connected layers to reduce number of features.
Args:
    inputs: A Tensor on which maxout will be performed
    num_units: Specifies how many features will remain after max pooling at the
      channel dimension. This must be multiple of number of channels.
    axis: The dimension where max pooling will be performed. Default is the
      last dimension.
    outputs_collections: The collections to which the outputs are added.
    scope: Optional scope for name_scope.
Returns:
    A `Tensor` representing the results of the pooling operation.
Raises:
    ValueError: if num_units is not multiple of number of features.
"""

def max_out(inputs, num_units, axis=None):
    shape = inputs.get_shape().as_list()
    if shape[0] is None:
        shape[0] = -1
    if axis is None:  # Assume that channel is the last dimension
        axis = -1
    num_channels = shape[axis]
    if num_channels % num_units:
        raise ValueError('number of features({}) is not '
                         'a multiple of num_units({})'.format(num_channels, num_units))
    shape[axis] = num_units
    shape += [num_channels // num_units]
    outputs = tf.reduce_max(tf.reshape(inputs, shape), -1, keep_dims=False)
    return outputs

another intuitive implementation:

def maxout(x, k, m):
    d = x.get_shape().as_list()[-1]

    W = tf.Variable(tf.random_normal(shape=[d, m, k]))
    b = tf.Variable(tf.random_normal(shape = [m, k]))
    z = tf.tensordot(x, W, axes=1) + b
    z = tf.reduce_max(z, axis=2)

    return z