[笔记][ProSeNet]Interpretable and steerable sequence learning via prototypes

Ming, Y., et al. (2019). Interpretable and steerable sequence learning via prototypes. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.

Methodology

We aim to learn representative prototype sequences (not necessarily exist in the training data) that can be used as classification references and analogical explanations.

D = { ( ( x ( t ) ) t = 1 T , y ) } \mathcal D=\{((x^{(t)})_{t=1}^T,y)\} D={((x(t))t=1T​,y)}: labeled sequence dataset

x ( t ) ∈ R n , y ∈ { 1 , . . . , C } x^{(t)}\in \R^n,y\in\{1,...,C\} x(t)∈Rn,y∈{1,...,C}

Architecture

$r\to p\to f$

$r$: sequence encoder

$p$: prototype layer

$f$: FC layer

sequence encoder

LSTM or GRU

$(x^{(t)}{)}_{t=1}^T\in {\mathbb{R}}^{n\times T}$: input

$e=h^{(T)}\in {\mathbb{R}}^m$: output, the hidden state at the last

prototype layer

$p_i\in {\mathbb{R}}^m$: prototype vectors 共k个

$a\in {\mathbb{R}}^k$: output $a_i=\exp (-||e-p_i|{|}_2^2)$

exp converts the distance to a score in $(0,1]$ (improvement)

FC layer

dim $k\to C$

weights $W$ non-negative

+softmax layer

$\hat{y}$: out

Loss func.

$$Loss(\Theta ,\mathcal{D})=CE(\Theta ,\mathcal{D})+{\lambda }_c{R}_c(\Theta ,\mathcal{D})+{\lambda }_e{R}_e(\Theta ,\mathcal{D})+{\lambda }_d{R}_d(\Theta ,\mathcal{D})+{\lambda }_{l_1}||W|{|}_1$$

$CE(\Theta ,\mathcal{D})$: cross entropy loss between $y$ and $\hat{y}$

$R_c(\Theta ,\mathcal{D})=\sum _{x^{(t)}{)}_{t=1}^T}\underset{i}{\min }||r((x^{(t)}{)}_{t=1}^T)-p_i|{|}_2^2$: item close to proto.

$R_e(\Theta ,\mathcal{D})=\sum _i\underset{x^{(t)}{)}_{t=1}^T}{\min }||r((x^{(t)}{)}_{t=1}^T)-p_i|{|}_2^2$: proto. close to item

$R_d(\Theta )=\sum _{i=1}^k\sum _{j=i+1}^k\max (0,d_{min}-||p_i-p_j|{|}_2{)}^2$: proto. not close

$d_{min}=1.0or2.0$ this model

$L_1$ penalty: help to learn sequence prototypes that have more unitary and additive semantics for classification

prototype projection

$p_i\leftarrow r(se{q}_i)$

$se{q}_i\leftarrow \arg \underset{seq\in r(\mathcal{X})}{\min }||seq-p_i|{|}_2$

simplification: 这里seq没有所有的sub-sequence遍历，而是使用beam search进行查找

The projection step is only performed every few training epochs (we set to 4 in our experiments) to reduce computational cost.

Refining ProSeNet with User Knowledge

set prototype mannually and no not update