《Hadoop: The Definitive Guide》读书笔记 -- Chapter 2 MapReduce

最新推荐文章于 2025-11-23 17:18:01 发布

原创最新推荐文章于 2025-11-23 17:18:01 发布 · 1.8k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#大数据 #hadoop

大数据专栏收录该内容

5 篇文章

订阅专栏

本文探讨了使用Hadoop MapReduce进行大规模数据处理的方法，包括如何将工作分割为等大小的部分、数据分布式存储、任务调度、数据局部优化、结果合并以及使用Hadoop Streaming进行文本处理的基本步骤。

Chapter 2 MapReduce

Analyzing the Data with Unix Tools

Problems

1. dividing work into equal-size isn't always easy or obvious.

2. Combining the results from independent processes may require further processing.

3. Still limited by the processing capacity of a single machine

Analyzing the Data with Hadoop

MapReduce

Scaling Out

To scale out, we need to store the data in a distributed filesystem(typically HDFS)

Tasks are scheduled by YARN.

Hadoop divides the input to a MapReduce job into fixed-size pieces called inputsplits. For most jobs, a good split size tends to be the size of anHDFS block, which is 128MB by default.

Data Locality Optimization: Hadoop does it best to run the map task on a node where the input data resides in HDFS. / Reduce task don't have the advantage of data locality.

Combiner reduce data transfered

Hadoop Streaming

Hadoop Streaming uses Unix standard streams as the interface between Hadoop and your program, so you can read standard input and write to standard output to write your MapReduce program. (Good for text processing)