MacOS Hadoop开发环境_mac hadoop streaming-优快云博客

本文链接：https://blog.youkuaiyun.com/qq_33096883/article/details/90749012

Hadoop

基本概念

Hadoop = HDFS + MapReduce. 大数据框架
HDFS : 分布式文件系统
MapReduce：大数据计算模型
NameNode：主节点；DataNode: 数据节点；SecondaryNameNode:备份
yarn：资源调度
- ResourceManager: 负责集群中所有资源的统一管理和分配
- NodeManager：管路Hadoop集群中单个计算节点

环境搭建

测试

map.py

import sys


def read_input(file):
    for line in file:
        yield line.split()


def main():
    data = read_input(sys.stdin)

    for words in data:
        for word in words:
            print("%s%s%d" % (word, "\t", 1))


if __name__ == '__main__':
    main()

reduce.py

import sys

from operator import itemgetter
from itertools import groupby


def read_map_out(file, sep='\t'):
    for line in file:
        yield line.rstrip().split(sep)


def main():
    data = read_map_out(sys.stdin)

    for cur_word, group in groupby(data, itemgetter(0)):
        tot_count = sum(int(count) for cur_word, count in group)
        print("%s%s%d"%(cur_word, "\t", tot_count))


if __name__ == '__main__':
    main()

本地测试echo "a b d e a v b" | python3 map.py | sort | python3 reduce.py
hdfs测试/usr/local/Cellar/hadoop/3.1.2/bin/hadoop jar /usr/local/Cellar/hadoop/3.1.2/libexec/share/hadoop/tools/lib/hadoop-streaming-3.1.2.jar -files "map.py,reduce.py" -input /kms.sh -output /output1 -mapper "/usr/local/bin/python3 map.py" -reducer "/usr/local/bin/python3 reduce.py"