Explainable ML

Explainable ML

Local Explanation

Why do you think this image is a cat?

判断录取、假释、金融相关的需要提供理由

模型诊断：到底学到了什么，不能只看正确率

Interpretable v.s. Powerful

Kaggle 大杀器：XGBoost

Object x -> Components: {x1, x2, …, xN}

Image: pixel, segment, etc.

Text: a word

Idea: Removing or modifying the values of the components, observing the change of decision

扰动：the size of the gray box can be crucial …

偏微分：|delta y| / |delta x|，saliency map，显著性分析，Gradient-based Approach

缺点：Gradient Saturation

解决办法：

• Integrated gradient
• DeepLIFT

加 noise 来 attack

pokemon vs Digimon，jpg、png

Global Explanation

What do you think a “cat” looks like?

Activation Maximization，理想的数字，Deep Neural Networks are easily fooled

$x^{\ast }=\arg {\max }_x{y}_i+R(x)$

“Regularization” from Generator

low-dim vector z -> Image Generator(GAN, VAE, etc.) G -> Image x

x = G(z)

Image Classifier != Image Discriminator

$x^{\ast }=\arg {\max }_x{y}_i\to z^{\ast }=\arg {\max }_z{y}_i$

$x^{\ast }=G\left(z^{\ast }\right)$

Generator 出来的人看得懂

arxiv：https://arxiv.org/pdf/1612.00005.pdf

与 GAN 不同，Image Generator 和 Image Classifier 固定不变

Using a model to explain another

Linear model cannot mimic neural network …,

Local Interpretable Model-Agnostic Explanations (LIME)

1. Given a data point you want to explain
2. Sample at the nearby
3. Fit with linear (or interpretable) model
4. Interpret the model you learned (using weight)

NLP(频谱图)？

Decision Tree 很深的话可以完全模仿 Black Box，

We don’t want the tree to be too large.

训练模型时提前考虑其要被 Decision Tree 解释：Tree regularization

${\theta }^{\ast }=\arg {\min }_{\theta }L(\theta )+\lambda O\left(T_{\theta }\right)$

无法微分？sol：arxiv：https://arxiv.org/pdf/1711.06178.pdf

Visual Analytics for Machine Learning

OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples

https://arxiv.org/abs/2002.03103

Data preperation -> Feature extraction -> Model selection -> Model training -> Evaluation -> Deployment

Solution = Data + ML Expertise + Computation

问题：

• 数据质量低
  • Poor Label
    • LabelInspect（TVCG 2018）
    • dataDebugger (VIS 2019)
  • Poor Coverage
    • OoDAnalyzer (TVCG)
  • Poor Amount
• 可解释性差
• 结果复杂、鲁棒性脆弱

Analyzing the Training Processes of Deep Generative Models

https://ieeexplore.ieee.org/document/8019879

Diagnosis Current techniques:

• Utilize the prediction score distributions of the model (i.e., sample-class probability) to evaluate the error severity.
• Utilize visual analytics to diagnose a failed training process.

Deep Generative Models: VAE, GAN, 引入了 Random variables

Training a DGM is Hard:

• DGM often involves both deterministic functions and random variables.
• DGM involves a top-down generative process and a bottom-up inference process

Challenges:

• Handle a large amount of time series data
  • Errors may arise from multiple possible sources: abnormal training samples, inappropriate network structures, and lack of numerical stability in the library
• Identify the root cause of a failed training process
  • it is often difficult to locate the specific neurons

Solution:

• A blue noise polyline sampling algorithm
• A credit assignment algorithm

Analytical process:

• Snapshot-level analysis
• Neuron-level analysis
• Layer-level analysis

Variational autoencoder: probabilistic version of an autoencoder

Analyzing the Noise Robustness of Deep Neural Networks

https://arxiv.org/abs/1810.03913

Motivation: Adversarial Examples

Challenges:

• Extract the datapath for adversarial examples
• Datapath visualization

A datapath

• Hundreds of layers
• Millions of neurons
• Millions of connections

Datapath Extraction - Quadratic Approximation

• Quadratic approximation
• Divide and conquer

Datapath visualization

• Euler-diagram-based layout to present feature maps in a layer
  • 分析两个集合之间的异同

A Survey of Visual Analytics Techniques for Machine Learning

Research Opportunities

https://arxiv.org/abs/2008.09632

• Opportunities before Model Building
  • Improving data quality for weakly supervised learning
  • Combine human expert knowledge and deep learning techniques through interactive visualization
• Opportunities in Model Building
  • Online training diagnosis
  • Interactive model refinement
• Opportunities after Model Building
  • Understanding multi-modal data
  • Analyzing concept drift for better performance
  • Improving the robustness of deep learning models for secure artificial intelligence
• A unified visual analytics framework for explaining deep learning models
  • 不是case by case，

目前可用的工具：tensorflow board

可视化大多数是 offline，不是 inline 的，如何 progressive 和 incremental 是很难的

toolkit

robust

模型的改进和调试

data-driven 和 knowledge-driven 如何结合

Reference

刘世霞_基于可视分析的可解释性人工智能

Visual Analytics for Explainable Deep Learning

Towards Better Analysis of Machine Learning Models: A Visual Analytics Perspective