wordcount程序

iteye_12567

于 2011-09-26 10:58:26 发布

阅读量149

点赞数

分类专栏： python 文章标签：大数据

python 专栏收录该内容

3 篇文章

订阅专栏

本文提供了一个简单的MapReduce实现示例，包括mapper和reducer函数。mapper函数负责将输入文本拆分为单词并计数，reducer函数汇总单词计数并输出。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import sys
def mapper(lines):
    maps=''
    lines.strip()
    lines=lines.split("\n")
    for line in lines:
        line.strip()
        words=line.split()
        for word in words:
            maps=maps+word+'\t1\n'
    return maps

def reducer(lines):
    wordCount={}
    lines.strip()    
    lines=lines.split("\n")
    lines=lines[0:len(lines)-2]
#    print lines
    for line in lines:
        line.strip()
        word,count=line.split('\t',1)
        try:
            count=int(count)
            wordCount[word]=wordCount.get(word,0)+count
        except ValueError:
            pass
    print wordCount.items()
    sorted_wordCount=sorted(wordCount.items(),key=lambda x:x[0])
    for word,count in sorted_wordCount:
        print("%s\t%s"%(word,count))
            

reducer(mapper("foo foo test show my mapp reduce quux labs\n"))

代码很容易改造成mapreduce形式在hadoop上运行，如http://blog.youkuaiyun.com/yaoyepeng/article/details/5929457