Chapter 3 N-gram Language Models

Chapter 3 N-gram Language Models

Speech and Language Processing ed3 读书笔记

Probabilities are essential in

• any task in which we have to identify words in noisy, ambiguous input, like speech recognition or handwriting recognition.
• spelling correction, which need to find and correct spelling errors
• machine translation
• augmentative communication systems

Models that assign probabilities to sequences of words are called language models or LMs. In this chapter we introduce the simplest model that assigns probabilities to sentences and sequences of words, the n-gram. This model is thus called n-gram model.

An n-gram is a sequence of N n-gram words: a 2-gram (or bigram) is a two-word sequence of words like “please turn”, “turn your”, or ”your homework”, and a 3-gram (or trigram) is a three-word sequence of words like “please turn your”, or “turn your homework”.

3.1 N-Grams

Let’s begin with the task of computing $P(w|h)$, the probability of a word $w$ given some history $h$. Suppose the history $h$ is “its water is so transparent that” and we want to know the probability that the next word is the:
$$P(the|itswaterissotransparentthat).$$
One way to estimate this probability is from relative frequency counts:
$$\begin{gathered} P(the|itswaterissotransparentthat)= \\ \frac{C(itswaterissotransparentthatthe)}{C(itswaterissotransparentthat)} \end{gathered}$$
Similarly, if we wanted to know the joint probability of an entire sequence of words like its water is so transparent, we could do it by asking “out of all possible sequences of five words, how many of them are its water is so transparent?” That seems rather a lot to estimate!

For this reason, we’ll need to introduce cleverer ways of estimating the probability of a word $w$ given a history $h$, or the probability of an entire word sequence $W$. Let’s start with a little formalizing of notation. To represent the probability of a particular random variable $X_i$ taking on the value “the”, or $P(X_i=“the”)$, we will use the simplification $P(the)$. We’ll represent a sequence of $N$ words either as $w_1\dots w_n$ or $w_1^n$ (so the expression $w_1^{n-1}$ means the string $w_1,w_2,\dots ,w_{n-1}$). For the joint probability of each word in a sequence having a particular value $P(X=w_1,Y=w_2,Z=w_3,\dots ,W=w_n)$ we’ll use $P(w_1,w_2,\dots ,w_n)$.

Now how can we compute probabilities of entire sequences like $P(w_1,w_2,\dots ,w_n)$? One thing we can do is decompose this probability using the chain rule of probability:
$$\begin{gathered} P(X_1{X}_2\dots X_n)=P(X_1)P(X_2|X_1)P(X_3|X_1^2)\dots P(X_n|X_1^{n-1}) \\ =\prod _{k=1}^{n}P(X_k|X_1^{k-1}) \end{gathered}$$
Applying the chain rule to words, we get
$$\begin{gathered} P(w_1^n)=P(w_1)P(w_2|w_1)P(w_3|w_1^2)\dots P(w_n|w_1^{n-1}) \\ =\prod _{k=1}^{n}P(w_k|w_1^{k-1}) \end{gathered}$$
The intuition of the n-gram model is that instead of computing the probability of a word given its entire history, we can approximate the history by just the last few words.

The bigram model, for example, approximates the probability of a word given all the previous words $P(w_n|w_1^{n-1})$ by using only the conditional probability of the preceding word $P(w_n|w_{n-1})$.
$$P(w_n|w_1^{n-1})\approx P(w_n|w_{n-1})$$
The assumption that the probability of a word depends only on the previous word is called a Markov assumption. The general equation for n-gram approximation to the conditional probability of the next word in a sequence is
$$P(w_n|w_1^{n-1})\approx P(w_n|w_{n-N+1}^{n-1})$$
Given the bigram assumption for the probability of an individual word, we can compute the probability of a complete word sequence
$$P(w_1^n)=\prod _{k=1}^{n}P(w_k|w_{k-1})$$
How do we estimate these bigram or n-gram probabilities? An intuitive way to estimate probabilities is called maximum likelihood estimation or MLE. We get the MLE estimate for the parameters of an n-gram model by getting counts from a normalize corpus, and normalizing the counts so that they lie between 0 and 1.

For example, to compute a particular bigram probability of a word $y$ given a previous word $x$, we’ll compute the count of the bigram $C(xy)$ and normalize by the sum of all the bigrams that share the same first word $x$:
$$P(w_n|w_{n-1})={\sum }_{w}C(w_{n-1}$$