language model

• ARPA format language model [slides, manpage, blog, htkbook]
  
  • Note: the log prob’s base is $10$
• http://www.statmt.org/book/slides/07-language-models.pdf
• 4-6 - Back-off and Interpolation - Stanford NLP - Dan Jurafsky & Chris Manning
• Lecture 7: Finite State Transducers, Language Modeling, and Speech Recognition Search

Back-off model:

$$S(w_i|w_{i-k+1}^{i-1})=\left\{ \begin{array}{lr} \frac{C(w^i_{i-k+1})}{C(w_{i-k+1}^{i-1})} & \text{if}\quad C(w_{i-k+1}^{i-1}) > 0\\ \alpha\cdot S(w_i|w_{i-k+2}^{i-1}) & \text{otherwise} \end{array}\right.$$

Toolkits:

• SRILM
• IRSTLM

Libraries:

• NGram

Create and manipulate n-gram language models encoded with weighted FSTs.

• Thrax

compile regular expressions and content-dependent rewrite grammars into weighted FSTs.