CS224N notes_chapter2_word2vec

第二讲 word2vec

1 Word meaning

the idea that is represented by a word, phrase, writing, art etc.
How do we have usable meaning in a computer?
Common answer: toxonomy(分类系统) like WordNet that has hypernyms relations(is-a) and synonym(同义词) sets.
Problems with toxonomy：

• missing nuances(细微差别) 比如 proficient 就比 good 更适合形容专家, 但是在分类系统中它们就是同义词
• missing new words
• subjective
• requires human labor to create and adapt
• Hard to compute accurate word similarity

Problems with discrete representation: one-hot representation dimensions.
[0,0,0,...,1,...,0]
and one-hot doesn’t give the relation/similarity between words.
Distributional similarity: you can get a lot of value for representing a word by means of its neighbors.
Next, we want to use vectors to represent words.
distributional: understand word meaning by context.
distributed:dense vectors to represent the meaning of the words.

2. Word2vec intro

Basic idea of learning Neural Network word embeddings
We def a model to predict the center word $w_t$ and context words in terms of word vectors.
$$p(context|w_t)$$
which has a loss function like
$$J=1-p(w_{-t}|w_t)$$
-t means neighbors of $w_t$ except $w_t$
Main idea of word2vec: Predict between every word and its context words.
Two algorithms.

1. Skip-grams(SG)
   Predict context words given target(position independent)
   … turning into banking crises as …
   banking: center word
   turning: $p(w_{t-2}|w_t)$
   For each word t=1,…T, we predict surrounding words in a window of “radius” m of every word
   $$\begin{gathered} J^{\prime }(\theta )=\prod _{t=1}^T\prod _{0m\le j\le m,j̸=0}P(w_{t+j}|w_t;\theta ) \\ J(\theta )=-\frac{1}{T}\sum _{t=1}^T\sum _{0m\le j\le m,j̸=0}P(w_{t+j}|w_t;\theta ) \end{gathered}$$
   hyperparameter: window size m
   we use $p(w_{t+j}|w_t)=\frac{exp(u_o^T{v}_c)}{{\sum }_{w=1}^{V}exp(u_w^T{v}_c)}$,
   the dot product will be greater if two words are more similar. And softmax maps the values to probability distribution.

2. Continuous Bag of Words(CBOW)
   Predict target word from bag-of-words context.

3. Research highlight

omit

4. Word2vec objective function gradients

all parameters in model
$$\theta =\left[\begin{array}{r}{v}_a\\ ⋮\\ v_{zebra}\\ u_a\\ ⋮\\ u_{zebra}\end{array}\right]$$
We try to optimize these parameters by training the model. We use gradients descent.
$$\begin{array}{rl}& \frac{\partial }{\partial v_c}\log exp(u_o^T{v}_c)-\log \sum _{x=1}^{V}exp(u_w^T{v}_c)\\ =& u_o-\frac{{\sum }_{x=1}^V{u}_{x}exp(u_x^T{v}_c)}{{\sum }_{w=1}^{V}exp(u_w^T{v}_c)}\\ =& u_0-\sum _{x=1}^{v}p(x|c)u_x\end{array}$$

5. Optimization refresher

We have the gradients at point x. Then we go along the negative gradients.
$${\theta }_j^{new}={\theta }_j^{old}-\alpha \frac{\partial }{\partial {\theta }_j^{old}}J(\theta )$$
$\alpha$: step size.
In matrix notation for parameters
$${\theta }_j^{new}={\theta }_j^{old}-\alpha {\nabla }_{\theta }J(\theta )$$
Stochastic Gradient Descent:

• global update -> much time
• mini batch -> also good idea

6. Assignment 1 notes

7. Usefulness of Wordvec