基于Gensim创建词向量

最新推荐文章于 2025-06-11 09:17:47 发布

原创最新推荐文章于 2025-06-11 09:17:47 发布 · 1.9k 阅读

1 ·

CC 4.0 BY-SA版权

人工智能专栏收录该内容

36 篇文章

订阅专栏

该博客介绍了如何利用Gensim库对文本数据进行预处理，去除停用词并进行分词，然后运用word2vec训练模型。训练完成后，保存并加载模型以进行相似词预测。示例中展示了对‘天龙八部.txt’文件处理后的结果，如‘武功’、‘弟子’等词汇的相似词。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

首先得对数据进行预处理。

去掉停用词以及结巴分词。将处理后的结果保存成文件。

本案使用的是天龙八部.txt

import jieba
import jieba.analyse
import  jieba.posseg as  pseg



stop_words = []
with open ('data/stopwords.txt','r',encoding='UTF-8') as  f:
    for line  in   f.readlines():
        line = line.replace("\n","").replace("\r","").strip()
        stop_words.append(line)


print(stop_words)


cut_result =   open ("data/after_cut.txt",'w',encoding='UTF-8')

def cut_words(filepath):
    with open (filepath,'r',encoding='UTF-8')as  f:

        for line in f.readlines():
            word_list = []

            line = line.replace("\n", "").replace("\r", "").strip()
            words  = jieba.lcut(line)
            for   word in words:
                if(word not in  stop_words):
                    word_list.append(word)
            cut_result.write(" ".join(word_list))
            cut_result.flush()





cut_words("data/天龙八部.txt")

然后用gensim中的word2vec训练预料。将训练后的模型保存，然后加载模型，对数据进行预测。

import logging
import os.path
import sys
import multiprocessing
from gensim.corpora import WikiCorpus
from gensim.models import Word2Vec
from gensim.models.word2vec import LineSentence


inp = "data/after_cut.txt"

# model = Word2Vec(LineSentence(inp), size=400, window=5, min_count=5, workers=multiprocessing.cpu_count())
#
# model.save("data/word2vec.model")



model = Word2Vec.load("data/word2vec.model")

test_word = ['丐帮','乔峰','大理','打狗棒']

for i in range(4):
    res =  model.most_similar(test_word[i])
    print(test_word[i])
    print(res)

执行效果如下：

/home/mengmeng/memgPro/ENV/bin/python /home/mengmeng/PycharmProjects/mengmeng/Gensim_01/__init__.py
/home/mengmeng/anaconda3/lib/python3.6/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
'See the migration notes for details: %s' % _MIGRATION_NOTES_URL
丐帮
[('武功', 0.9950281381607056), ('弟子', 0.994763970375061), ('昆仑', 0.9946858882904053), ('伏牛', 0.9923895597457886), ('蓬莱', 0.9909208416938782), ('签条', 0.9904066920280457), ('门人', 0.9900406002998352), ('帮主', 0.9900399446487427), ('五袋', 0.9887819290161133), ('门下', 0.9886124134063721)]
乔峰
[('玄苦', 0.999525249004364), ('乔帮主', 0.9994806051254272), ('帮规', 0.9994643926620483), ('找', 0.9994380474090576), ('听说', 0.999426007270813), ('位', 0.999393880367279), ('师弟', 0.9993889331817627), ('这事', 0.999376118183136), ('帮', 0.9993749856948853), ('害死', 0.9993677139282227)]
大理
[('国', 0.9945481419563293), ('段氏', 0.9917622208595276), ('段家', 0.9895750880241394), ('国人', 0.9895458221435547), ('兄弟', 0.9889448881149292), ('天龙', 0.988686740398407), ('玄悲大师', 0.9884297251701355), ('今日', 0.9879704117774963), ('南疆', 0.9879286289215088), ('皇帝', 0.9879129528999329)]
打狗棒
[('成', 0.9998838901519775), ('号令', 0.9998779296875), ('子弟', 0.999876856803894), ('本帮', 0.9998766779899597), ('率领', 0.9998753070831299), ('久', 0.9998719096183777), ('一套', 0.999871551990509), ('相助', 0.9998712539672852), ('种种', 0.9998687505722046), ('经书', 0.9998682141304016)]

Process finished with exit code 0