[笔记][Transformer]Attention Is All You Need_vaswani a, shazeer n, parmar n, et al. attention i-优快云博客

本文链接：https://blog.youkuaiyun.com/weixin_44834206/article/details/123448597

transformer 原论文 Attention is all you need 阅读笔记
建议结合原文食用

Reference：
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.

圈圈 2022/3/12 初稿结合代码介绍了模型
以下为正文↓

Attention Is All You Need

Transformer

model architecture

$(x_1,...,x_n)\xrightarrow{encoder} (z_1,...,z_n)\xrightarrow{decoder} (y_1,...,y_m)$

encoder and decoder stacks

encoder

composed of a stack of $N = 6$ identical layers, each layer has two sub-layers

multi-head self-attention mechanism $x\to S(x)$

residue connection followed by layer normalization
$x\to Layernorm(x+S(x))$
simple position-wise fully connected feed-forward network +
same residue connection followed by layer normalization

To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension $d_{model}=512$

## code author: "Yu-Hsiang Huang", https://github.com/jadore801120/attention-is-all-you-need-pytorch


class EncoderLayer(nn.Module):
    ''' Compose with two layers '''

    def __init__(self, d_model, d_inner, n_head, d_k, d_v, dropout=0.1):
        super(EncoderLayer, self).__init__()
        self.slf_attn = MultiHeadAttention(n_head, d_model, d_k, d_v, dropout=dropout)
        self.pos_ffn = PositionwiseFeedForward(d_model, d_inner, dropout=dropout)

    def forward(self, enc_input, slf_attn_mask=None):
        enc_output, enc_slf_attn = self.slf_attn(
            enc_input, enc_input, enc_input, mask=slf_attn_mask)
        enc_output = self.pos_ffn(enc_output)
        return enc_output, enc_slf_attn

decoder

also composed of a stack of $N = 6$ identical layers, but has three sub-layers each

the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack.

在这里插入图片描述

class DecoderLayer(nn.Module):
    ''' Compose with three layers '''

    def __init__(self, d_model, d_inner, n_head, d_k, d_v, dropout=0.1):
        super(DecoderLayer, self).__init__()
        self.slf_attn = MultiHeadAttention(n_head, d_model, d_k, d_v, dropout=dropout)
        self.enc_attn = MultiHeadAttention(n_head, d_model, d_k, d_v, dropout=dropout)
        self.pos_ffn = PositionwiseFeedForward(d_model, d_inner, dropout=dropout)

    def forward(
            self, dec_input, enc_output,
            slf_attn_mask=None, dec_enc_attn_mask=None):
        dec_output, dec_slf_attn = self.slf_attn(
            dec_input, dec_input, dec_input, mask=slf_attn_mask)
        dec_output, dec_enc_attn = self.enc_attn(
            dec_output, enc_output, enc_output, mask=dec_enc_attn_mask)
        dec_output = self.pos_ffn(dec_output)
        return dec_output, dec_slf_attn, dec_enc_attn

attention

An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.

scaled dot-product attention

Q : packed queries query dim : $d_k$
K : packed keys key dim : $d_k$
V : packed values value dim : $d_v$

$Attention(Q,K,V)=softmax(\dfrac{QK^T}{\sqrt{d_k}})V$

reason for scaling the dot products by $\frac{1}{\sqrt{d_k}}$ :

While for small values of dk the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of $d_k$ . We suspect that for large values of $d_k$ , the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients.

在这里插入图片描述

multi-head attention

several attention layers in parallel + projection
$MultiHead(Q,K,V)=Concat(head_1,...,head_h)W^O \\ where\ head_i=Attention(QW^Q_i,KW^K_i,VW^V_i)\\ projection\ matrices:\ W^Q_i\in\mathbb R^{d_{model}\times d_k},\\ \ W^K_i\in\mathbb R^{d_{model}\times d_k},\ W^V_i\in\mathbb R^{d_{model}\times d_v},\ W^O\in\mathbb R^{hd_v\times d_{model}}$

position-wise feed-forward networks

$FFN(x)=ReLU(xW_1+b_1)W_2+b_2$

2 full-connect layers

self.w_1 = nn.Linear(d_in, d_hid) # position-wise
self.w_2 = nn.Linear(d_hid, d_in) # position-wise

x = self.w_2(F.relu(self.w_1(x)))

embeddings and softmax

share the same weight matrix between the two embedding layers and the pre-softmax linear transformation

         self.trg_word_prj = nn.Linear(d_model, n_trg_vocab, bias=False)
    
         seq_logit = self.trg_word_prj(dec_output)
         if self.scale_prj:
            seq_logit *= self.d_model ** -0.5

softmax $\to$ probability

positional encoding

Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence.

$PE_{(pos,2i)}=\sin(pos/10000^{2i/d_{model}})\\ PE_{(pos,2i+1)}=\cos(pos/10000^{2i/d_{model}})$

sinusoidal version, of course can use other version

        sinusoid_table[:, 0::2] = np.sin(sinusoid_table[:, 0::2])  # dim 2i
    
        sinusoid_table[:, 1::2] = np.cos(sinusoid_table[:, 1::2])  # dim 2i+1