2017-12-26-gensim-word2vec-优快云博客

本文介绍了一种流行的词向量模型Word2Vec的基本训练方法，包括简易示例和大规模语料上的训练技巧。通过示例代码展示了如何使用Gensim库进行Word2Vec模型的训练、保存与加载，并提供了相似词查询、词语关系推理等实用功能。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

title

tags

date

5.Word2Vec

nlp

post

Gensim

nlp

2017-12-26 20:56:24

最简单的word2vec训练方式：

# import modules & set up logging
import gensim
#build for sentences list
sentences = [['first', 'sentence'], ['second', 'sentence']]
# train word2vec on the two sentences
model = gensim.models.Word2Vec(sentences, min_count=1,workers=4)

如果文本太多，可以使用一种优化内存的加载方法：

#build from files
import os

class MemorySentences(object):
    def __init__(self,path):
        self.path=path
    def __iter__(self):
        for filename in os.listdir(self.path):
            for line in open(os.path.join(self.path,filename),'r'):
                yield list(line)
corpus=MemorySentences('../data/wiki_zh/')
model = gensim.models.Word2Vec(corpus, min_count=1,workers=4)

模型的保存和加载：

#model save and load
model.save('/tmp/mymodel')
new_model = gensim.models.Word2Vec.load('/tmp/mymodel')
model = gensim.models.Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False)
# using gzipped/bz2 input works too, no need to unzip:
model = gensim.models.Word2Vec.load_word2vec_format('/tmp/vectors.bin.gz', binary=True)

使用方法：

#usage
model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)
#[('queen', 0.50882536)]
model.doesnt_match("breakfast cereal dinner lunch".split())
'cereal'
model.similarity('woman', 'man')
#0.73723527