Wordcount -- MapReduce example -- Reducer

最新推荐文章于 2025-05-12 11:04:25 发布

转载最新推荐文章于 2025-05-12 11:04:25 发布 · 50 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/LexLuc/p/9571033.html

文章标签：

#大数据

本文详细介绍了MapReduce中Reducer的工作原理及其实现方式。Reducer接收来自Mapper的键值对，并将这些值聚合为所需格式，最终将产生的键值对写回HDFS。文中还提供了Reducer类的具体实现代码示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Reducer receives (key, values) pairs and aggregate values to a desired format, then write produced (key, value) pairs back into HDFS.
E.g.
Input: (term, [1, 1, 1, 1])
Output: (term, 4)

Reducer Class Prototype:

Reducer<Text, IntWritable, Text, IntWritable> 
// Text:: INPUT_KEY
// IntWritable:: INPUT_VALUE
// Text:: OUTPUT_KEY
// IntWritable:: OUTPUT_VALUE

Reduce Method for Mapper

Method header

public void reduce(Text key, Iterable<IntWritable> values,
                     Context context
                     ) throws IOException, InterruptedException 
// Text key:: Declare data type of input key;
// Iterable<IntWritable> values:: Declare data type of input values; (Note: Received values from mapper should be in a list)
// Context context:: Declare data type of output. Context is often used for output data collection.

Aggregate Values

// Iterate through all the values wrt the key:
int sum = 0;
for (IntWritable val : values) {
  sum += val.get();
}

Building (key, value) pairs

// Convert built-in int into IntWritable
result.set(sum);
// build (key, value) pair into Context and emit:
context.write(key, result);

Reducer Class Summary

Reducer class produces Reducer.Context object and serialize obtained (key, value) pair into HDFS.

Overview of Reducer Class

public static class IntSumReducer
     extends Reducer<Text,IntWritable,Text,IntWritable> {
  private IntWritable result = new IntWritable();

  public void reduce(Text key, Iterable<IntWritable> values,
                     Context context
                     ) throws IOException, InterruptedException {
    int sum = 0;
    for (IntWritable val : values) {
      sum += val.get();
    }
    result.set(sum);
    context.write(key, result);
  }
}

Written with StackEdit.

转载于:https://www.cnblogs.com/LexLuc/p/9571033.html