CMU 11-785 L04 Backpropagation

Problem setup

• Input-output pairs: not to mention

• Representing the output: one-hot vector

  • $$y_i=\frac{\exp \left(z_i\right)}{{\sum }_j\exp \left(z_j\right)}$$

  • two classes of softmax = sigmoid

• Divergence: must be differentiable

  • For real-valued output vectors, the (scaled) $L_2$ divergence

    • $$\mathrm{Div}(Y,d)=\frac{1}{2}\Vert Y-d{\Vert }^2=\frac{1}{2}\sum _i{\left(y_i-d_i\right)}^2$$

  • For binary classifier

    • $$\mathrm{Div}(Y,d)=-\mathrm{dlog}Y-(1-d)\log (1-Y)$$

    • Note: the derivative is not zero even $d=Y$, but it can converge very quickly

  • For multi-class classification

    • $$\mathrm{Div}(Y,d)=-\sum _i{d}_i\log y_i=-\log y_c$$

    • If $y_c<1$ , the slope is negative w.r.t. $y_c$, indicates increasing $y_c$ will reduce divergence

Train the network

Distributed Chain rule

$$y=f\left(g_1(x),g_1(x),\dots ,g_M(x)\right)$$

$$\frac{dy}{dx}=\frac{\partial f}{\partial g_1(x)}\frac{d{g}_1(x)}{dx}+\frac{\partial f}{\partial g_2(x)}\frac{d{g}_2(x)}{dx}+\cdots +\frac{\partial f}{\partial g_M(x)}\frac{d{g}_M(x)}{dx}$$

Backpropagation

• For each layer: we caculate $\frac{\partial Div}{\partial y_i}$,$\frac{\partial Dicv}{\partial z}$, and $\frac{\partial Div}{\partial w_{ij}}$

• For ouput layer

  • It is easy to caculate $\frac{\partial Div}{\partial y_i^{(N)}}$
  • So: $\frac{\partial Div}{\partial z_i^{(N)}}=f_N^{\prime }\left(z_i^{(N)}\right)\frac{\partial Div}{\partial y_i^{(N)}}$
  • $\frac{\partial Div}{\partial w_{ij}^{(N)}}=\frac{\partial z_j^{(N)}}{\partial w_{ij}^{(N)}}\frac{\partial Div}{\partial z_j^{(N)}}$, where $\frac{\partial z_j^{(N)}}{\partial w_{ij}^{(N)}}=y_i^{(N)}$

• Pass on

  • $z_j^{(N)}={\sum }_i{w}_{ij}^{(2)}{y}_i^{(v-1)}$, so $\frac{\partial z_j^{(N)}}{\partial y_1^{(N-1)}}=w_{ij}^{(N)}$
  • $\frac{\partial Div}{\partial y_i^{(N-1)}}={\sum }_j{w}_{ij}^{(N)}\frac{\partial Div}{\partial z_j^{(N)}}$
  • $\frac{\partial Div}{\partial z_i^{(N-1)}}=f_{N-1}^{\prime }(z_i^{(N-1)})\frac{\partial Div}{\partial y_i^{(N-1)}}$
  • $\frac{\partial Div}{\partial w_{ij}^{(N-1)}}=y_i^{(N-2)}\frac{\partial Div}{\partial z_j^{(N-1)}}$

Special case

Vector activations

• Vector activations: all outputs are functions of all inputs

• So the derivatives need to change a little

• $$\frac{\partial Div}{\partial z_i^{(k)}}=\sum _j\frac{\partial Div}{\partial y_j^{(k)}}\frac{\partial y_j^{(k)}}{\partial z_i^{(k)}}$$

• Note: derivatives of scalar activations are just a special case of vector activations:

• $$\frac{\partial y_j^{(k)}}{\partial z_i^{(k)}}=0\text{ for }i\ne j$$

• For example, Softmax:

$$y_i^{(k)}=\frac{\exp \left(z_i^{(k)}\right)}{{\sum }_j\exp \left(z_j^{(k)}\right)}$$

$$\frac{\partial Div}{\partial z_i^{(k)}}=\sum _j\frac{\partial Div}{\partial y_j^{(k)}}\frac{\partial y_j^{(k)}}{\partial z_i^{(k)}}$$

KaTeX parse error: Got function '\newline' with no arguments as argument to '\left' at position 1: \̲n̲e̲w̲l̲i̲n̲e̲

• Using Keonecker delta ${\delta }_{ij}=1$ if $i=j,0$ if $i\ne j$

$$\frac{\partial Div}{\partial z_i^{(k)}}=\sum _j\frac{\partial Div}{\partial y_j^{(k)}}{y}_i^{(k)}\left({\delta }_{ij}-y_j^{(k)}\right)$$

Multiplicative networks

• Some types of networks have multiplicative combination(instead of additive combination)
• Seen in networks such as LSTMs, GRUs, attention models, etc.

• So the derivatives need to change

$$\frac{\partial Div}{\partial o_i^{(k)}}=\sum _j{w}_{ij}^{(k+1)}\frac{\partial Div}{\partial z_j^{(k+1)}}$$

$$\frac{\partial Div}{\partial y_j^{(k-1)}}=\frac{\partial o_i^{(k)}}{\partial y_j^{(k-1)}}\frac{\partial Div}{\partial o_i^{(k)}}=y_l^{(k-1)}\frac{\partial Div}{\partial o_i^{(k)}}$$

• A layer of multiplicative combination is a special case of vector activation

Non-differentiable activations

• Activation functions are sometimes not actually differentiable

  • The RELU (Rectified Linear Unit)
    • And its variants: leaky RELU, randomized leaky RELU
  • The “max” function

• Subgradient

  • $$\left(f(x)-f\left(x_0\right)\right)\ge v^T\left(x-x_0\right)$$

  • The subgradient is a direction in which the function is guaranteed to increase

  • If the function is differentiable at $x$ , the subgradient is the gradient

  • But gradient is not always the subgradient though

Vector formulation

• Define the vectors:

Forward pass

Backward pass

• Chain rule
  • $\mathbf{y}=\mathbf{f}(\mathbf{g}(\mathbf{x}))$
  • Let $z=g(x)$,$y=f(z)$
  • So $J_{\mathbf{y}}(\mathbf{x})=J_{\mathbf{y}}(\mathbf{z})J_{\mathbf{z}}(\mathbf{x})$
• For scalar functions:
  • $D=f(Wy+b)$
  • Let $z=Wy+b$, $D=f(z)$
  • ${\nabla }_{x}D={\nabla }_z(D)J_z(x)$
• So for backward process
  • ${\nabla }_{Z_N}Div={\nabla }_{Y}Div{\nabla }_{Z_N}Y$
  • $\nabla_{y_{N-1}}Div = \nabla_{Z_N} Div \nabla_{y_{N-1}} z_N $
  • ${\nabla }_{W_N}Div=y_{N-1}{\nabla }_{Z_N}Div$
  • ${\nabla }_{b_N}Div={\nabla }_{Z_N}Div$
• For each layer
  • First compute ${\nabla }_{y}Div$
  • Then compute ${\nabla }_{z}Div$
  • Finally ${\nabla }_{W}Div$, ${\nabla }_{W}Div$

Training

Analogy to forward pass