Understanding the difficulty of training deep feedforward neural networks

softsign

A newly proposed activation function (Bergstra et al., 2009) called the $softsign$:

$$x/(1+|x|)$$
The $softsign$ is similar to the hyperbolic tangent but its tails are quadratic polynomials rather than exponentials, i.e., it approaches its asymptoes much slower.

normalized initialization

$$W \sim U \left[ -\frac{\sqrt 6}{\sqrt {n_j+n_{j+1}}}, \frac{\sqrt 6}{\sqrt {n_j+n_{j+1}}} \right]$$