自然语言处理 - Language model和RNN

本文翻译和精简自stanford cs224n lec 6.

1. Language Model

通俗的说，language model就是用来预测下一个出现的词的概率，即：
$$P(x^{(t+1)}|x^{(t)},x^{(t-1)},...x^{(1)})$$

1.1 统计学方法：n-gram language model

简化：一个词出现的概率只和它前面的n-1个词有关系，这就是"n-gram"的含义。因此有:
P ( x ( t + 1 ) ∣ x ( t ) , x ( t − 1 ) , . . . x ( 1 ) ) = P ( x ( t + 1 ) ∣ x ( t ) , x ( t − 1 ) , . . . x ( t − n + 2 ) ) = P ( x ( t + 1 ) , x ( t ) , x ( t − 1 ) , . . . x ( t − n + 2 ) ) P ( x ( t ) , x ( t − 1 ) , . . . x ( t − n + 2 ) ) = c o u n t ( x ( t + 1 ) , x ( t ) , x ( t − 1 ) , . . . x ( t − n + 2 ) ) c o u n t ( x ( t ) , x ( t − 1 ) , . . . x ( t − n + 2 ) ) P(x^{(t+1)}|x^{(t)},x^{(t-1)},...x^{(1)}) =P(x^{(t+1)}|x^{(t)},x^{(t-1)},...x^{(t-n+2)}) \\ = \frac{P(x^{(t+1)},x^{(t)},x^{(t-1)},...x^{(t-n+2)})}{P(x^{(t)},x^{(t-1)},...x^{(t-n+2)})} \\ = \frac{count(x^{(t+1)},x^{(t)},x^{(t-1)},...x^{(t-n+2)})}{count(x^{(t)},x^{(t-1)},...x^{(t-n+2)})} P(x(t+1)∣x(t),x(t−1),...x(1))=P(x(t+1)∣x(t),x