Lecture 12: Visualizing and Understanding

CS231n

Lecture 12: Visualizing and Understanding

Visualize

• Filters of first layer
• feature sof last layer: t-SNE
• Visualizing Activations
• Occlusion Experiments
• Saliency Maps: 梯度反传到输入

• Intermediate features via guided backprop: grad *= relu(y)

  • Pick a single intermediate neuron
  • Compute gradient of neuron value with respect to image pixels

  Images come out nicer if you only backprop positive gradients through each ReLU
  Find the part of an image that a neuron responds to
  对象是输入图像，需要给定

• Visualizing CNN features: Gradient Ascent
  Generate a synthetic image that maximally activates a neuron, 对象是模型本身，无需给定输入图像
  

  $$I^*=\arg\max_I f(I)+R(I) \Rightarrow \arg\max_I S_c(I)-\lambda\lVert I \rVert^2_2$$
  1. Initialize image to zeros
  2. Forward image to compute current scores
  3. Backprop to get gradient of neuron value with respect to image pixels
  4. Make a small update to the image

• Adversarial Examples, 可以给定输入图像
  

  $$x_\mathrm{adv} = x + \arg\min_\delta \lVert \delta \rVert \quad \mathrm{s.t.} \mathrm{Pred}(x_\mathrm{adv}) \ne \mathrm{Pred}(x)$$
  DeepDream, 可以给定输入图像
  1. Forward: compute activations at chosen layer
  2. Set gradient of chosen layer equal to its activation
  3. Backward: Compute gradient on image
  4. Update image
• Feature Inversion, 由特征找回原图像
  Given a CNN feature vector for an image, find a new image that:
  
  • Matches the given feature vector
  • “looks natural” (image prior regularization)
    $$x^*= \arg\min_x \quad l(\Phi(x),\Phi_0) + \lambda R(x) = \arg\min_x\quad \lVert \Phi(x) - \Phi_0 \rVert^2 + \lambda \int\lVert \nabla f \rVert^\beta \mathrm{d}r$$
• Texture Synthesis: Gram matrix $\Rightarrow$ Neural Style Transfer
  原有nueral style方法对于一张内容图像和风格图像迭代生成太慢$\Rightarrow$Fast Style Transfer: 训练一个专门的模型生成某一特定的风格图像（训练网络使生成的图像满足传统neural style中的目标，或者对图像进行多尺度融合直接生成相应风格的图像）$\Rightarrow$单个模型生成多种风格（学习每一种风格的scale和shift，选定一组scale和shift后用同一个模型生成相应风格的图像）