大数据hadoop系列：python实现MapReduce 词频统计

map代码：map_t.py

import sys
import re

p = re.compile(r'\w+')
for line in sys.stdin:
    ss = line.strip().split(' ')
    for s in ss:
        if len(p.findall(s)) < 1:
            continue
        s_low = p.findall(s)[0].lower()
        print s_low + ',' + '1'

reduce代码：red_t.py

import sys

cur_word = None
sum = 0
for line in sys.stdin:
    word, val = line.strip().split(',')
    if cur_word == None:
        cur_word = word
    if cur_word != word:
        print '%s\t%s' % (cur_word, sum)
        cur_word = word
        sum = 0
    sum += int(val)
print '%s\t%s' % (cur_word, sum)

测试shell：

cat The_Man_of_Property.txt | python map_t.py |sort -k2|python red_t.py

run.sh

<