Attention Is All You Need

Attention Is All You Need

Abstract

任务：机器翻译
传统方法：RNN/CNN，可能加上attention机制
本文的方法：Transformer（变形金刚？变压器？），只用attention

Introduction

RNN’s inherent sequential nature precludes parallelization within training examples.
attention mechanisms are used in conjunction with a recurrent network
Transformer: relying entirely on an attention mechanism to draw global dependencies between input and output

Background

Self-attention（SAGAN中介绍过）: intra-attention, relating different positions of a single sequence in order to compute a representation of the sequence

Model Architecture

encoder-decoder structure
encoder: $(x_1,...,x_n)\mapsto(z_1,...,z_n)$
auto-regressive: the output variable depends linearly on its own previous values and on a stochastic term (an imperfectly predictable term)

$$X_t = c + \sum_{i = 1}^p \phi_iX_{t-i} + \epsilon_t$$

Scaled Dot-Product Attention: queries $Q$, keys $K$, values $V$, dimension of keys $d_k$,

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^\mathrm{T}}{\sqrt{d_k}})V$$

Multi-Head Attention: concatenate, project

$$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat(head_1,...,head}_h)W^O\\ \mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K,VW_i^V)$$

fully connected feed-forward network: 两层全连接网络

$$\mathrm{FFN}(x) = \max(0,xW_1+b_1)W_2+b_2$$

Positional Encoding: make use of the order of the sequence, relative or absolute position of the tokens in the sequence.

$$PE(pos,2i) = \sin(\frac{pos}{10^{4\cdot \frac{2i}{d_\mathrm{model}}}})\\ PE(pos,2i+1) = \cos(\frac{pos}{10^{4\cdot \frac{2i}{d_\mathrm{model}}}})$$
allow the model to easily learn to attend by relative positions

Why Self-Attention

1. the total computational complexity per layer
2. the amount of computation that can be parallelized
3. the path length between long-range dependencies in the network
4. yield more interpretable models