文件方式实现完整的英文词频统计实例

最新推荐文章于 2020-11-20 22:09:52 发布

转载最新推荐文章于 2020-11-20 22:09:52 发布 · 123 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/honghui/p/7598447.html

本文通过一个简单的童谣示例，展示了如何进行基本的文本分析过程，包括读取文本、去除标点符号、转换为小写、分词、排除常用停用词、统计词频并输出前20个最频繁出现的词汇。

1.读入待分析的字符串

2.分解提取单词

3.计数字典

4.排除语法型词汇

5.排序

6.输出TOP(20)

fo=open('123.txt','w')
fo.write('''Twinkle, twinkle, little star, How I wonder what you are.
Up above the world so high, Like a diamond in the sky. 
Twinkle, twinkle, little star, How I wonder what you are!
When the blazing sun is gone, 
When he nothing shines upon, 
Then you show your little light, 
Twinkle, twinkle, all the night. 
Twinkle, twinkle, little star, 
How I wonder what you are! 
Then the traveler in the dark Thanks you for your tiny spark;
He could not see which way to go, If you did not twinkle so.
Twinkle, twinkle, little star, How I wonder what you are!
Twinkle Twinkle Little Star''')
fo.close()


fo =open('123.txt','r')
A= fo.read()
exc={'the','and','to','of','in','a','for','with',''}
for i in ',.?!\n"':
    A=A.replace(i,' ')
A=A.lower()
A=A.split(" ")
words=set(A)
dic={}
keys=set(A)#出现过单词的集合，字典的KEY
keys=keys-exc
for i in keys:
    dic[i]=A.count(i)
w=list(dic.items())
w.sort(key=lambda x:x[1],reverse=True)
for i in range(20):
    print(w[i])
fo.close()