CMU 11-785 L19 Hopfield network

Hopfield Net

• So far, neural networks for computation are all feedforward structures

Loopy network

• Each neuron is a perceptron with +1/-1 output
  • Every neuron receives input from every other neuron
  • Every neuron outputs signals to every other neuron

• At each time each neuron receives a “field” ${\sum }_{j\ne i}{w}_{ji}{y}_j+b_i$
  • If the sign of the field matches its own sign, it does not respond
  • If the sign of the field opposes its own sign, it “flips” to match the sign of the field

• If the sign of the field at any neuron opposes its own sign, it “flips” to match the field
  • Which will change the field at other nodes
  • Which may then flip… and so on…

Filp behavior

• Let $y_i^{-}$ be the output of the $i$-th neuron just before it responds to the current field

• Let $y_i^{+}$ be the output of the $i$-th neuron just after it responds to the current field

• if $y_i^{-}=\mathrm{sign}\left({\sum }_{j\ne i}{w}_{ji}{y}_j+b_i\right)$, then $y_i^{+}=-y_i^{-}$

  • If the sign of the field matches its own sign, it does not flip

  • $$y_i^{+}\left(\sum _{j\ne i}{w}_{ji}{y}_j+b_i\right)-y_i^{-}\left(\sum _{j\ne i}{w}_{ji}{y}_j+b_i\right)=0$$

• if $y_i^{-}\ne \mathrm{sign}\left({\sum }_{j\ne i}{w}_{ji}{y}_j+b_i\right)$, then $y_i^{+}=-y_i^{-}$

  • $$y_i^{+}\left(\sum _{j\ne i}{w}_{ji}{y}_j+b_i\right)-y_i^{-}\left(\sum _{j\ne i}{w}_{ji}{y}_j+b_i\right)=2y_i^{+}\left(\sum _{j\ne i}{w}_{ji}{y}_j+b_i\right)$$

  • This term is always positive!

• Every flip of a neuron is guaranteed to locally increase $y_i\left({\sum }_{j\ne i}{w}_{ji}{y}_j+b_i\right)$

Globally

• Consider the following sum across all nodes

$$\begin{array}{c}D\left(y_1,y_2,\dots ,y_N\right)={\sum }_i{y}_i\left({\sum }_{j\ne i}{w}_{ji}{y}_j+b_i\right)\\ \\ ={\sum }_{i,j\ne i}{w}_{ij}{y}_i{y}_j+{\sum }_i{b}_i{y}_i\end{array}$$

• Assume $w_{ii}=0$
• For any unit $k$ that “flips” because of the local field

$$\Delta D\left(y_k\right)=D\left(y_1,\dots ,y_k^{+},\dots ,y_N\right)-D\left(y_1,\dots ,y_k^{-},\dots ,y_N\right)$$

$$\Delta D\left(y_k\right)=\left(y_k^{+}-y_k^{-}\right)\left(\sum _{j\ne k}{w}_{jk}{y}_j+b_k\right)$$

• This is always positive!
• Every flip of a unit results in an increase in $D$

Overall

• Flipping a unit will result in an increase (non-decrease) of

$$D=\sum _{i,j\ne i}{w}_{ij}{y}_i{y}_j+\sum _i{b}_i{y}_i$$

• $D$ is bounded

$$D_{\max }=\sum _{i,j\ne i}\left|w_{ij}\right|+\sum _i\left|b_i\right|$$

• The minimum increment of $D$ in a flip is

ΔDmin⁡=min⁡i,{yi,i=1.…N}2∣∑j≠iwjiyj+bi∣ \Delta D_{\min }=\min _{i,\{y_{i}, i=1 . \ldots N\}} 2|\sum_{j \neq i} w_{j i} y_{j}+b_{i}| ΔDmin​=i,{yi​,i=1.…N}min​2∣j​=i∑​wji​yj​+bi​∣

• Any sequence of flips must converge in a finite number of steps
  • Think of this as an infinite deep network where every weights at every layers are identical
  • Find the maximum layer!

The Energy of a Hopfield Net

• Define the Energy of the network as

$$E=-\sum _{i,j\ne i}{w}_{ij}{y}_i{y}_j-\sum _i{b}_i{y}_i$$

• Just the negative of $D$

• The evolution of a Hopfield network constantly decreases its energy

• This is analogous to the potential energy of a spin glass(Magnetic diploes)

  • The system will evolve until the energy hits a local minimum

• We remove bias for better understanding
  

• The network will evolve until it arrives at a local minimum in the energy contour

Content-addressable memory

• Each of the minima is a “stored” pattern
  • If the network is initialized close to a stored pattern, it will inevitably evolve to the pattern
• This is a content addressable memory
  • Recall memory content from partial or corrupt values
• Also called associative memory
  • Evolve and recall pattern by content, not by location

Evolution

• The network will evolve until it arrives at a local minimum in the energy contour
• We proved that every change in the network will result in decrease in energy
• So path to energy minimum is monotonic

For 2-neuron net

• Symmetric
  • $-\frac{1}{2}{\mathbf{y}}^T\mathbf{W}\mathbf{y}=-\frac{1}{2}(-\mathbf{y}{)}^T\mathbf{W}(-\mathbf{y})$
  • If $\hat{y}$ is a local minimum, so is $-\hat{y}$

Computational algorithm

• Very simple
• Updates can be done sequentially, or all at once
• Convergence when it deos not chage significantly any more

$$E=-\sum _i\sum _{j>i}{w}_{ji}{y}_j{y}_i$$

Issues

Store a specific pattern

• A network can store multiple patterns
  • Every stable point is a stored pattern
  • So we could design the net to store multiple patterns
    • Remember that every stored pattern $P$ is actually two stored patterns, $P$ and $-P$
• How could the quadrtic function have multiple minimum? (Convex function)
  • Input has constrain (belong to $(-1,1)$ )
• Hebbian learning: $w_{ji}=y_j{y}_i$
• Design a stationary pattern
  • $\mathrm{sign}\left({\sum }_{j\ne i}{w}_{ji}{y}_j\right)=y_i\forall i$
• So
  • $\mathrm{sign}\left({\sum }_{j\ne i}{w}_{ji}{y}_j\right)=\mathrm{sign}\left({\sum }_{j\ne i}{y}_j{y}_i{y}_j\right)$
  • $=\mathrm{sign}\left({\sum }_{j\ne i}{y}_j^2{y}_i\right)=\mathrm{sign}\left(y_i\right)=y_i$
• Energy
  • $\begin{array}{rl}E=& -\sum _i\sum _{j<i}{w}_{ji}{y}_j{y}_i=-\sum _i\sum _{j<i}{y}_i^2{y}_j^2\\ & \\ & =-\sum _i\sum _{j<i}1=-0.5N(N-1)\end{array}$
  • This is the lowest possible energy value for the network

• Stored pattern has lowest energy
• No matter where it begin, it will evolve into yellow pattern(lowest energy)

How many patterns can we store?

• To store more than one pattern

wji=∑yp∈{yp}yipyjp w_{j i}=\sum_{\mathbf{y}_{p} \in\left\{\mathbf{y}_{p}\right\}} y_{i}^{p} y_{j}^{p} wji​=yp​∈{yp​}∑​yip​yjp​

• {yP}\{y_P\}{yP​} is the set of patterns to store
• Super/subscript $p$ represents the specific pattern
• Hopfield: For a network of neurons can store up to ~$0.15N$ patterns through Hebbian learning(Provided in PPT)

Orthogonal/ Non-orthogonal patterns

• Orthogonal patterns
  

  • Patterns are local minima (stationary and stable)
    • No other local minima exist
    • But patterns perfectly confusable for recall

• Non-orthogonal patterns
  

  • Patterns are local minima (stationary and stable)
    • No other local minima exist
      • Actual wells for patterns
    • Patterns may be perfectly recalled! (Note K > 0.14 N)

• Two orthogonal 6-bit patterns
  

  • Perfectly stationary and stable
  • Several spurious “fake-memory” local minima…

Observations

• Many “parasitic” patterns

  • Undesired patterns that also become stable or attractors

• Patterns that are non-orthogonal easier to remember

  • I.e. patterns that are closer are easier to remember than patterns that are farther!!

• Seems possible to store K > 0.14N patterns

  • i.e. obtain a weight matrix W such that K > 0.14N patterns are stationary
  • Possible to make more than 0.14N patterns at-least 1-bit stable