综合练习：词频统计

最新推荐文章于 2019-04-27 23:32:10 发布

weixin_33962621

最新推荐文章于 2019-04-27 23:32:10 发布

阅读量108

点赞数

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/2015110114z/p/8649854.html

本文介绍了一种处理英文文本的方法，包括去除标点符号、转换大小写、统计词频并排除常见语法词汇的过程。最终输出了词频最高的前20个单词。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

综合练习

词频统计预处理

下载一首英文的歌词或文章

将所有,.？！’:等分隔符全部替换为空格

将所有大写转换为小写

生成单词列表

生成词频统计

排序

排除语法型词汇，代词、冠词、连词

输出词频最大TOP20

text='''
I became what I am today at the age of twelve, on a frigid overcast day in the winter of 1975. I remember the precise moment, crouching behind a crumbling mud wall, peeking into the alley near the frozen creek. That was a long time ago, but it's wrong what they say about the past, I've learned, about how you can bury it. Because the past claws its way out. Looking back now, I realize I have been peeking into that deserted alley for the last twenty-six years.
One day last summer, my friend Rahim Khan called from Pakistan. He asked me to come see him. Standing in the kitchen with the receiver to my ear, I knew it wasn't just Rahim Khan on the line. It was my past of unatoned sins. After I hung up, I went for a walk along Spreckels Lake on the northern edge of Golden Gate Park. The early-afternoon sun sparkled on the water where dozens of miniature boats sailed, propelled by a crisp breeze. Then I glanced up and saw a pair of kites, red with long blue tails, soaring in the sky. They danced high above the trees on the west end of the park, over the windmills, floating side by side like a pair of eyes looking down on San Francisco, the city I now call Home. And suddenly Hassan's voice whispered in my head: _For you, a thousand times over_. Hassan the harelipped kite runner.
I sat on a park bench near a willow tree. I thought about something Rahim Khan said just before he hung up, almost as an after thought. _There is a way to be good again_. I looked up at those twin kites. I thought about Hassan. Thought about Baba. Ali. Kabul. I thought of the life I had lived until the winter of 1975 came and changed everything. And made me what I am today.

'''
#标点替换为空格
symbol='''!?,.@#$%*_+-'''
# 把文章的标点符号替换
for i in symbol:
    text=text.replace(i,'')
# 改成小写  以空格将字符串分成单词列表
textlist=text.lower().split()

#用字典记录单词和其出现次数
dic={}
for i in textlist:
    count=text.count(i)
    dic[i]=count

words = '''a an the in on to at and of is was are were i he she you your they us their our it or for be too do no 
that s so as but it's'''

for i in words:
    if(dic.get(i)!=None): #如果为冠词之类的无意义的词，将其舍弃
        dic.pop(i)

new_dic = sorted(dic.items(),key=lambda x:x[1],reverse = True)

for i in range(20):
    print(new_dic[i]) #取出现频率最高的10个单词

print(dic)

将分析对象存为utf-8编码的文件，通过文件读取的方式获得词频分析内容。

fo = open('text.txt', 'r')
text=fo.read()
fo.close()

转载于:https://www.cnblogs.com/2015110114z/p/8649854.html