手写MapReduce

最新推荐文章于 2022-06-11 16:22:23 发布

Colin_lqk

最新推荐文章于 2022-06-11 16:22:23 发布

阅读量274

点赞数

分类专栏： Colin_lqk的大数据之路文章标签：大数据 mapreduce hadoop java

本文链接：https://blog.youkuaiyun.com/weixin_45250512/article/details/117049573

版权

本文通过详细步骤介绍如何手写一个MapReduce程序，包括Mapper、Reducer和Driver的实现。在Mapper阶段，数据被拆分成键值对；Reducer阶段对Mapper输出进行聚合处理；Driver则是整个流程的控制器，负责运行任务。此外，还提及了项目的依赖管理文件pom.xml。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

MapperDemo

package com.work;

/**
 * Created by Colin_li on 2021-05-19 23:44
 */
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;


public class MapperDemo extends Mapper<LongWritable, Text, LongWritable, NullWritable> {
   

    LongWritable outKey = new LongWritable();
    NullWritable outValue = NullWritable.get();

    /**
     * 按行处理数据源
     * 一行是数字，在mapper的过程中把Text类型的输入数据转换为LongWritable作为mapper的输出key,
     * 借助shuffle过程中的默认自动排序来实现需求中的数据整体排序
     * @param key
     * @param value
     * @param context
     * @throws IOException
     * @throws InterruptedException
     */
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
   
        long res =  Long.parseLong(value.toString());
        outKey.set(res);
        context.write(outKey, outValue);
    }
}

ReducerDemo

package com.work;

/**
 * Created by Colin_li on 2021-05-19 23:44
 */
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;